docs/build.tex

   1 One of the biggest challenges to getting started with embedded devices is that you
   2 just can't install a copy of Linux and expect to be able to compile a firmware.
   3 Even if you did remember to install a compiler and every development tool offered,
   4 you still wouldn't have the basic set of tools needed to produce a firmware image.
   5 The embedded device represents an entirely new hardware platform, which is
   6 incompatible with the hardware on your development machine, so in a process called
   7 cross compiling you need to produce a new compiler capable of generating code for
   8 your embedded platform, and then use it to compile a basic Linux distribution to
   9 run on your device.
  10
  11 The process of creating a cross compiler can be tricky, it's not something that's
  12 regularly attempted and so the there's a certain amount of mystery and black magic
  13 associated with it. In many cases when you're dealing with embedded devices you'll
  14 be provided with a binary copy of a compiler and basic libraries rather than
  15 instructions for creating your own -- it's a time saving step but at the same time
  16 often means you'll be using a rather dated set. Likewise, it's also common to be
  17 provided with a patched copy of the Linux kernel from the board or chip vendor,
  18 but this is also dated and it can be difficult to spot exactly what has been
  19 changed to make the kernel run on the embedded platform.
  20
  21 \subsection{Building an image}
  22
  23 OpenWrt takes a different approach to building a firmware, downloading, patching
  24 and compiling everything from scratch, including the cross compiler. Or to put it
  25 in simpler terms, OpenWrt doesn't contain any executables or even sources, it's an
  26 automated system for downloading the sources, patching them to work with the given
  27 platform and compiling them correctly for the platform. What this means is that
  28 just by changing the template, you can change any step in the process.
  29
  30
  31 As an example, if a new kernel is released, a simple change to one of the Makefiles
  32 will download the latest kernel, patch it to run on the embedded platform and produce
  33 a new firmware image -- there's no work to be done trying to track down an unmodified
  34 copy of the existing kernel to see what changes had been made, the patches are
  35 already provided and the process ends up almost completely transparent. This doesn't
  36 just apply to the kernel, but to anything included with OpenWrt -- It's this one
  37 simple understated concept which is what allows OpenWrt to stay on the bleeding edge
  38 with the latest compilers, latest kernels and latest applications.
  39
  40 So let's take a look at OpenWrt and see how this all works
  41
  42
  43 \subsubsection{Download openwrt}
  44
  45 This article refers to the "Kamikaze" branch of OpenWrt, which can be downloaded via
  46 subversion using the following command:
  47
  48 \begin{Verbatim}
  49 svn co https://svn.openwrt.org/openwrt/trunk kamikaze
  50 \end{Verbatim}
  51
  52 Additionally, there's a trac interface on \href{https://dev.openwrt.org/}{https://dev.openwrt.org/}
  53 which can be used to monitor svn commits and browse the sources.
  54
  55
  56 \subsubsection{The directory structure}
  57
  58 There are four key directories in the base:
  59
  60 \begin{itemize}
  61         \item tools
  62         \item toolchain
  63         \item package
  64         \item target
  65 \end{itemize}
  66
  67 \texttt{tools} and \texttt{toolchain} refer to common tools which will be
  68 used to build the firmware image and the compiler and c library.
  69 The result of this is three new directories, \texttt{tool\_build}, which is a temporary
  70 directory for building the target independent tools, \texttt{toolchain\_build\_\textit{<arch>}}
  71 which is used for building the toolchain for a specific architecture, and
  72 \texttt{staging\_dir\_\textit{<arch>}} where the resulting toolchain is installed.
  73 You won't need to do anything with the toolchain directory unless you intend to
  74 add a new version of one of the components above.
  75
  76 \texttt{package} is for exactly that -- packages. In an OpenWrt firmware, almost everything
  77 is an \texttt{.ipk}, a software package which can be added to the firmware to provide new
  78 features or removed to save space.
  79
  80 \texttt{target} refers to the embedded platform, this contains items which are specific to
  81 a specific embedded platform. Of particular interest here is the "\texttt{target/linux}"
  82 directory which is broken down by platform and contains the kernel config and patches
  83 to the kernel for a particular platform. There's also the "\texttt{target/image}" directory
  84 which describes how to package a firmware for a specific platform.
  85
  86 Both the target and package steps will use the directory "\texttt{build\_\textit{<arch>}}"
  87 as a temporary directory for compiling. Additionally, anything downloaded by the toolchain,
  88 target or package steps will be placed in the "\texttt{dl}" directory.
  89
  90
  91 \subsubsection{Building OpenWrt}
  92
  93 While the OpenWrt build environment was intended mostly for developers, it also has to be
  94 simple enough that an inexperienced end user can easily build his or her own customized firmware.
  95
  96 Running the command "\texttt{make menuconfig}" will bring up OpenWrt's configuration menu
  97 screen, through this menu you can select which platform you're targeting, which versions of
  98 the toolchain you want to use to build and what packages you want to install into the
  99 firmware image. Similar to the linux kernel config, almost every option has three choices,
 100 \texttt{y/m/n} which are represented as follows:
 101
 102 \begin{itemize}
 103         \item{\texttt{<*>} (pressing y)} \\
 104                 This will be included in the firmware image
 105         \item{\texttt{<M>} (pressing m)} \\
 106                 This will be compiled but not included (for later install)
 107         \item{\texttt{< >} (pressing n)} \\
 108                 This will not be compiled
 109 \end{itemize}
 110
 111 After you've finished with the menu configuration, exit and when prompted, save your
 112 configuration changes. To begin compiling the firmware, type "\texttt{make}". By default
 113 OpenWrt will only display a high level overview of the compile process and not each individual
 114 command.
 115
 116 \subsubsection{Example:}
 117
 118 \begin{Verbatim}
 119 make[2] toolchain/install
 120 make[3] -C toolchain install
 121 make[2] target/compile
 122 make[3] -C target compile
 123 make[4] -C target/utils prepare
 124
 125 [...]
 126 \end{Verbatim}
 127
 128 This makes it easier to monitor which step it's actually compiling and reduces the amount
 129 of noise caused by the compile output. To see the full output, run the command
 130 "\texttt{make V=99}".
 131
 132 During the build process, buildroot will download all sources to the "\texttt{dl}"
 133 directory and will start patching and compiling them in the "\texttt{build\_\textit{<arch>}}"
 134 directory. When finished, the resulting firmware will be in the "\texttt{bin}" directory
 135 and packages will be in the "\texttt{bin/packages}" directory.
 136
 137
 138 \subsection{Creating packages}
 139
 140
 141 One of the things that we've attempted to do with OpenWrt's template system is make it
 142 incredibly easy to port software to OpenWrt. If you look at a typical package directory
 143 in OpenWrt you'll find two things:
 144
 145 \begin{itemize}
 146         \item \texttt{package/\textit{<name>}/Makefile}
 147         \item \texttt{package/\textit{<name>}/patches}
 148 \end{itemize}
 149
 150 The patches directory is optional and typically contains bug fixes or optimizations to
 151 reduce the size of the executable. The package makefile is the important item, provides
 152 the steps actually needed to download and compile the package.
 153
 154 Looking at one of the package makefiles, you'd hardly recognize it as a makefile.
 155 Through what can only be described as blatant disregard and abuse of the traditional
 156 make format, the makefile has been transformed into an object oriented template which
 157 simplifies the entire ordeal.
 158
 159 Here for example, is \texttt{package/bridge/Makefile}:
 160
 161 \begin{Verbatim}[frame=single,numbers=left]
 162 include $(TOPDIR)/rules.mk
 163
 164 PKG_NAME:=bridge
 165 PKG_VERSION:=1.0.6
 166 PKG_RELEASE:=1
 167
 168 PKG_BUILD_DIR:=$(BUILD_DIR)/bridge-utils-$(PKG_VERSION)
 169 PKG_SOURCE:=bridge-utils-$(PKG_VERSION).tar.gz
 170 PKG_SOURCE_URL:=@SF/bridge
 171 PKG_MD5SUM:=9b7dc52656f5cbec846a7ba3299f73bd
 172 PKG_CAT:=zcat
 173
 174 include $(INCLUDE_DIR)/package.mk
 175
 176 define Package/bridge
 177   SECTION:=base
 178   CATEGORY:=Network
 179   DEFAULT:=y
 180   TITLE:=Ethernet bridging configuration utility
 181   URL:=http://bridge.sourceforge.net/
 182 endef
 183
 184 define Package/bridge/description
 185 Ethernet bridging configuration utility
 186   Manage ethernet bridging; a way to connect networks together
 187   to form a larger network.
 188 endef
 189
 190 define Build/Configure
 191   $(call Build/Configure/Default, \
 192     --with-linux-headers=$(LINUX_DIR))
 193 endef
 194
 195 define Package/bridge/install
 196         install -m0755 -d $(1)/usr/sbin
 197         install -m0755 $(PKG_BUILD_DIR)/brctl/brctl \
 198                 $(1)/usr/sbin/
 199 endef
 200
 201 $(eval $(call BuildPackage,bridge))
 202 \end{Verbatim}
 203
 204
 205 As you can see, there's not much work to be done; everything is hidden in other makefiles
 206 and abstracted to the point where you only need to specify a few variables.
 207
 208 \begin{itemize}
 209         \item \texttt{PKG\_NAME} \\
 210                 The name of the package, as seen via menuconfig and ipkg
 211         \item \texttt{PKG\_VERSION} \\
 212                 The upstream version number that we're downloading
 213         \item \texttt{PKG\_RELEASE} \\
 214                 The version of this package Makefile
 215         \item \texttt{PKG\_BUILD\_DIR} \\
 216                 Where to compile the package
 217         \item \texttt{PKG\_SOURCE} \\
 218                 The filename of the original sources
 219         \item \texttt{PKG\_SOURCE\_URL} \\
 220                 Where to download the sources from
 221         \item \texttt{PKG\_MD5SUM} \\
 222                 A checksum to validate the download
 223         \item \texttt{PKG\_CAT} \\
 224                 How to decompress the sources (zcat, bzcat, unzip)
 225 \end{itemize}
 226
 227 The \texttt{PKG\_*} variables define where to download the package from;
 228 \texttt{@SF} is a special keyword for downloading packages from sourceforge.
 229 The md5sum is used to verify the package was downloaded correctly and
 230 \texttt{PKG\_BUILD\_DIR} defines where to find the package after the sources are
 231 uncompressed into \texttt{\$(BUILD\_DIR)}.
 232
 233 At the bottom of the file is where the real magic happens, "BuildPackage" is a macro
 234 setup by the earlier include statements. BuildPackage only takes one argument directly --
 235 the name of the package to be built, in this case "\texttt{bridge}". All other information
 236 is taken from the define blocks. This is a way of providing a level of verbosity, it's
 237 inherently clear what the contents of the \texttt{description} template in
 238 \texttt{Package/bridge} is, which wouldn't be the case if we passed this information
 239 directly as the Nth argument to \texttt{BuildPackage}.
 240
 241 \texttt{BuildPackage} uses the following defines:
 242
 243 \textbf{\texttt{Package/\textit{<name>}}:} \\
 244         \texttt{\textit{<name>}} matches the argument passed to buildroot, this describes
 245         the package the menuconfig and ipkg entries. Within \texttt{Package/\textit{<name>}}
 246         you can define the following variables:
 247
 248         \begin{itemize}
 249                 \item \texttt{SECTION} \\
 250                         The type of package (currently unused)
 251                 \item \texttt{CATEGORY} \\
 252                         Which menu it appears in menuconfig
 253                 \item \texttt{TITLE} \\
 254                         A short description of the package
 255                 \item \texttt{URL} \\
 256                         Where to find the original software
 257                 \item \texttt{MAINTAINER} (optional) \\
 258                         Who to contact concerning the package
 259                 \item \texttt{DEPENDS} (optional) \\
 260                         Which packages must be built/installed before this package
 261         \end{itemize}
 262
 263 \textbf{\texttt{Package/\textit{<name>}/conffiles} (optional):} \\
 264    A list of config files installed by this package, one file per line.
 265
 266 \textbf{\texttt{Build/Prepare} (optional):} \\
 267    A set of commands to unpack and patch the sources. You may safely leave this
 268    undefined.
 269
 270 \textbf{\texttt{Build/Configure} (optional):} \\
 271    You can leave this undefined if the source doesn't use configure or has a
 272    normal config script, otherwise you can put your own commands here or use
 273    "\texttt{\$(call Build/Configure/Default,\textit{<args>})}" as above to
 274    pass in additional arguments for a standard configure script.
 275
 276 \textbf{\texttt{Build/Compile} (optional):} \\
 277    How to compile the source; in most cases you should leave this undefined.
 278
 279 \textbf{\texttt{Package/\textit{<name>}/install}:} \\
 280    A set of commands to copy files out of the compiled source and into the ipkg
 281    which is represented by the \texttt{\$(1)} directory.
 282
 283 The reason that some of the defines are prefixed by "\texttt{Package/\textit{<name>}}"
 284 and others are simply "\texttt{Build}" is because of the possibility of generating
 285 multiple packages from a single source. OpenWrt works under the assumption of one
 286 source per package makefile, but you can split that source into as many packages as
 287 desired. Since you only need to compile the sources once, there's one global set of
 288 "\texttt{Build}" defines, but you can add as many "Package/<name>" defines as you want
 289 by adding extra calls to \texttt{BuildPackage} -- see the dropbear package for an example.
 290
 291 After you've created your \texttt{package/\textit{<name>}/Makefile}, the new package
 292 will automatically show in the menu the next time you run "make menuconfig" and if selected
 293 will be built automatically the next time "\texttt{make}" is run.
 294
 295 \subsubsection{Troubleshooting}
 296
 297 If you find your package doesn't show up in menuconfig, try the following command to
 298 see if you get the correct description:
 299
 300 \begin{Verbatim}
 301   TOPDIR=$PWD make -C package/<name> DUMP=1 V=99
 302 \end{Verbatim}
 303
 304 If you're just having trouble getting your package to compile, there's a few
 305 shortcuts you can take. Instead of waiting for make to get to your package, you can
 306 run one of the following:
 307
 308 \begin{itemize}
 309         \item \texttt{make package/\textit{<name>}-clean V=99}
 310         \item \texttt{make package/\textit{<name>}-install V=99}
 311 \end{itemize}
 312
 313 Another nice trick is that if the source directory under \texttt{build\_\textit{<arch>}}
 314 is newer than the package directory, it won't clobber it by unpacking the sources again.
 315 If you were working on a patch you could simply edit the sources under the
 316 \texttt{build\_\textit{<arch>}/\textit{<source>}} directory and run the install command above,
 317 when satisfied, copy the patched sources elsewhere and diff them with the unpatched
 318 sources. A warning though - if you go modify anything under \texttt{package/\textit{<name>}}
 319 it will remove the old sources and unpack a fresh copy.
 320