A camera is unnecessary when an air-conditioner controller only needs a small vocabulary of hand motions. The useful signal is not an image of the hand; it is a directional event such as up, down, left, or right. A short-range optical gesture sensor can reduce that input to a few bytes before the ESP32-S3 sees it.

That changes the embedded design substantially. There is no camera frame buffer, DVP bus, image preprocessing, or inference model competing with the LCD and infrared transmitter. The ESP32-S3 can spend its resources on the part that actually needs careful handling: keeping the intended AC state synchronized with each gesture and encoding that state into the appliance’s IR protocol.

The gesture sensor replaces the vision pipeline

The APDS-9960 combines proximity, ambient-light, color, and gesture sensing behind an I2C interface. Its gesture engine uses four directional photodiodes and an integrated IR emitter to sense directional movement close to the device.

For a controller mounted on a wall or desk, the data path can stay small:

hand movement
     |
     v
APDS-9960
     | I2C
     v
ESP32-S3
  |       |
  |       +--> LCD
  |
  +----------> IR LED --> AC

The sensor does not reconstruct a hand pose. It reports information from which simple directional gestures can be recognized. That distinction matters. This design is suitable for deliberate swipes near the controller, not finger counting, sign-language recognition, or room-scale hand tracking.

Broadcom specifies the APDS-9960 as supporting simple up, down, right, and left gestures. Its proximity function is factory-calibrated around a short 100 mm range. That short operating distance is useful here: a person has to approach the controller intentionally, which reduces accidental commands from ordinary movement elsewhere in the room.

The APDS-9960 itself is now marked obsolete by Broadcom, so a new product should also evaluate an actively supported gesture or proximity part. The architecture does not depend on this exact sensor; it depends on receiving a small, explicit gesture event rather than a camera frame.

Four gestures are enough for the control surface

A compact mapping can cover the requested AC functions:

Gesture Action
Up Power on
Down Power off
Left Select COOL
Right Select DRY

The mapping should be treated as an interface contract rather than scattered conditionals. The firmware first converts raw sensor output into a gesture event, then passes that event to the AC state machine.

enum class Gesture {
  None,
  Up,
  Down,
  Left,
  Right
};

enum class AcMode {
  Cool,
  Dry
};

struct AcState {
  bool power;
  AcMode mode;
  uint8_t temperature;
};

A gesture changes AcState; it does not directly toggle GPIO pins for the appliance.

AC infrared control is stateful

Television remotes often make people think of infrared as one code per button. Air-conditioner remotes frequently behave differently. Many AC protocols transmit a representation of the remote’s current state, including properties such as power, operating mode, target temperature, fan speed, and swing settings.

That means a gesture such as “select DRY” should be interpreted as a state transition:

before:
power = ON
mode  = COOL
temp  = 24 C

gesture:
RIGHT

after:
power = ON
mode  = DRY
temp  = 24 C

then:
encode complete supported AC state
send IR frame

This is safer than treating COOL and DRY as unrelated raw pulse arrays. If another gesture later changes power or temperature, the firmware still knows what state it intends the appliance to have.

The IRremoteESP8266 project exposes this model explicitly through IRac and stdAc::state_t for supported air-conditioner protocols. Protocol-specific classes are available when the common abstraction cannot express a feature. Support still depends on the actual AC brand, remote, protocol, and sometimes model variant.

A small state machine prevents ambiguous gestures

Gesture sensors can produce repeated or incomplete observations while a hand crosses the sensing area. Sending IR immediately for every intermediate observation can turn one swipe into several appliance commands.

Put a command boundary between recognition and transmission:

IDLE
  |
  | valid gesture
  v
UPDATE_STATE
  |
  v
SEND_IR
  |
  v
UPDATE_LCD
  |
  v
COOLDOWN
  |
  | hand leaves / timeout expires
  v
IDLE

The cooldown is not an AC protocol requirement. It is an input-policy mechanism that prevents a single physical gesture from being interpreted repeatedly.

A practical handler can keep that policy separate from protocol encoding:

void applyGesture(Gesture gesture, AcState &state) {
  switch (gesture) {
    case Gesture::Up:
      state.power = true;
      break;

    case Gesture::Down:
      state.power = false;
      break;

    case Gesture::Left:
      state.power = true;
      state.mode = AcMode::Cool;
      break;

    case Gesture::Right:
      state.power = true;
      state.mode = AcMode::Dry;
      break;

    default:
      return;
  }

  sendAcState(state);
  renderAcState(state);
}

The exact decision to turn the unit on when selecting a mode is a product choice. Keeping it in one transition function makes that behavior visible and testable.

The LCD should display intended state, not pretend to be feedback

The LCD can show room temperature, selected AC mode, and power state without carrying camera video:

26.4 C

AC: ON
MODE: COOL
SET: 24 C

There is an important limitation: a normal IR transmitter is one-way. Updating the LCD after sending a frame proves that the ESP32-S3 attempted the command; it does not prove that the air conditioner received or applied it.

Someone can also use the original remote and change the appliance without the ESP32-S3 knowing. The displayed value is therefore the controller’s intended state unless the AC provides a separate feedback channel or the system adds a receiver capable of observing relevant remote traffic.

That semantic distinction prevents a polished interface from presenting an assumption as measured appliance state.

Temperature sensing belongs on a temperature sensor

The controller may also display room temperature or use it for automation. That measurement should come from a temperature sensor, not from the gesture device.

An I2C temperature and humidity sensor can share the control bus when addresses and electrical requirements permit. The resulting architecture remains modest in GPIO use:

I2C
 +-- gesture sensor
 +-- temperature/humidity sensor

SPI or parallel display bus
 +-- LCD

GPIO
 +-- IR transmitter driver

If the LCD uses a parallel interface, removing the camera is especially valuable. An OV2640-style DVP camera consumes an 8-bit pixel bus plus clocks and synchronization signals. A gesture sensor moves the human input to I2C and leaves substantially more GPIO and peripheral bandwidth available for the display.

The IR LED needs a proper driver

The ESP32-S3 should generate the modulated IR signal, but a GPIO is not a substitute for an IR LED driver stage. The optical range depends on LED current, pulse conditions, emitter characteristics, orientation, and the AC receiver.

A typical hardware boundary is:

ESP32-S3 GPIO
      |
      v
transistor / MOSFET driver
      |
      v
    IR LED
      |
      v
AC IR receiver

The resistor, transistor, and LED current must be selected from the actual component ratings. Copying an arbitrary resistor value from a schematic without checking supply voltage, forward voltage, allowed pulse current, and driver characteristics can produce either weak range or excessive current.

Capture the real remote before assuming protocol compatibility

The controller cannot infer an AC protocol from the fact that the appliance has an infrared remote. Before fixing the firmware architecture around a protocol, identify the exact AC and remote and inspect captured transmissions.

IRremoteESP8266 documents support for many AC protocol families and recommends capture tools such as IRrecvDumpV2 when identifying a remote. If the protocol has detailed library support, the firmware can construct stateful commands. If it does not, raw replay can reproduce known states, but transitions become harder to maintain because each capture represents a particular remote configuration.

For this project, the useful capture set is small but deliberate:

OFF
ON + COOL + chosen temperature
ON + DRY + chosen temperature

Additional captures are needed when fan, swing, temperature, or other settings are exposed in the interface.

Camera-free gesture control reduces the problem to deterministic events

For ON, OFF, COOL, and DRY, computer vision adds little value. The controller needs four intentional events, not a visual model of the user’s hand.

A short-range gesture sensor turns those movements into compact input, the ESP32-S3 applies them to a persistent AC state, the IR layer encodes that state for the actual appliance, and the LCD presents what the controller intends. The difficult boundary is no longer image recognition. It is keeping gesture events, AC protocol state, and displayed state consistent even when infrared itself provides no acknowledgement.

ESP32-S3 Gesture Sensor Infrared Air Conditioner Embedded Systems