Well to "build" a handheld of your own, you have to make some decisions with regards to the microprocessors, RAM, ROM, etc. you want to use. If you want to 'build" those from scratch, then you have to look into FPGA and learn a HDL like VHDL.
Once you have the h/w, then you can use C/C++ and assembly to code the OS.
Things for you to look up on Wikipedia:
FPGA
VHDL
Microprocessor
Microcontroller
C++
Assembly
RTOS (well you don't need a "real time" OS, but the topic RTOS will talk about the needed elements - e.g. scheduler)
I am sure the other guys will list more stuff
