A 32MB disk-on-chip memory can be found at US$20 at surplus stores. For some applications, that's more than enough. For more capacity and frequent rewriting, a small capacitor hard disk (IDE, 30GB) is about US$50. Furthermore, must Mini-ITX boards now come with USB 2.0 interfaces. This means you can use a US$50 1GB jump drive as your mass storage.
Mini-ITX motherboards can use regular ATX power supplies. However, one can also use smaller power supplies. A kit consisting of a 55W 12VDC DC-DC converter and a 110-230 VAC to 12VDC power supply is about US$60.
By the time you have a fully functional computer, you probably will have spent about US$220 to US$300.