citizen428.blog()

Try to learn something about everything

Information Overload 2014-09-28

Information Overload 2014-09-21

After an almost 2 year hiatus, it’s time to revive this series, although I’m not yet sure if it’s going to be weekly again.

Re-kindling My Love Affair With the Kindle Fire

When I originally bought my Kindle Fire HD 7”, I was madly in love with it. However, after the initial euphoria was over, I got very annoyed with the dumbed down user interface and the limited availability of apps in Amazon’s app store.

So yesterday morning I finally set aside around 2 hours to do the following:

All in all the process went pretty smoothly and I now I have a mostly uncrippled Android tablet. There are some small glitches, but they are certainly less annoying than having a very limited device. At first I couldn’t get the Kindle app to run (it’s a modified version that’s pretty tied to the Fire’s native launcher), but after a bit of fiddling around it’s now working. Some people also opt for not selecting any launcher as default action for the home button, so they can easily switch between the Kindle experience and a more tablet like launcher.

If you own a Kindle Fire that you don’t use much maybe you should give rooting a try, it may re-kindle your interest in the device.

VCR Case Woes

Today I spent quite some time on a rather hard to track down problem. I had some specs using VCR, which worked perfectly locally, but just wouldn’t run on CircleCI. In fact, it seemed like CircleCI didn’t in fact use the VCR cassettes at all, but instead make a real API call, which failed because the necessary credentials weren’t available in the environment (which was kinda the point of using VCR in the first place).

Turns out the problem lay in the combination of using the VCR setting configure_rspec_metadata! and the option core.ignorecase = true, which is standard in newer versions of Git. What happened was that at one point I changed the case of a word in a test description, which changes the auto-generated name of the folder VCR will create the cassette in. But since the folder name wasn’t changed from git’s perspective, I never realized that. So when the code was checked out on CircleCI, VCR encountered the spec with the metadata (vcr: true), but couldn’t find the cassette since the checked out folder had the wrong case. It therefore proceeded to run the spec against the live API which failed.

All in all this cost me about 1.5 hours, and I only managed to find a solution thanks to a helpful colleague and CircleCI’s option of letting you SSH into the machine the tests run on, which is how we discovered the second folder that was created.

TIL that git’s not case-sensitive. I’d rather have found out another way.

2013 Reading List

Same post as every year, here are the book I read last year. However, this time around there are several novelties: first off, my Spanish got good enough to actually read books, so there are 3 languages now (English, German, Spanish). Also my Kindle Fire allowed me to get back into comics, so I (re-)read a lot of the classics and great stuff I missed out on over the years. Last but not least this is also the year where I started listening to audiobooks, mostly because they are really practical on long bus journeys because they keep you occupied but you can still enjoy the landscape.

The links are affiliate links by the way, I have to finance my reading habit somehow ;–)

Books:

Comics:

Audiobooks:

Around Southeast Asia

Still trying to catch up on updates, this one will cover the time from end of August until late October. The countries: Malaysia, Thailand, Cambodia.

While we were in Vietnam, we realized we really felt like some beach (that was before we actually got get quite a bit of that in Vietnam itself). WWe also remembered that we really liked Malaysia and would definitely go back there, so we bought a flight to KL with the idea of visiting some places we missed last time and later going to Perhentian or Tioman island. KL was chaotic as usual, but I still kinda like the place. It’s nothing special, but we have a guesthouse where we always stay and some restaurants we really like. Sometimes returning to the familiar feels good when you are on a long trip… But because we also felt like something new we headed down to Melakka, which although rather touristy is a really nice place, with good food, a relaxed atmosphere and lots of pretty old houses in the Straigths Chinese style. Definitely recommended! Next stop: Georgetown. Two years ago we already spent a couple of days on Penang island, but we never really bothered with visiting its main city. Turns out that was a great mistake, because I’m absolutely in love with this place now! There’s the usual mix of cultures typical for Malaysia, although Georgetown’s population is primarily of Indian and Chinese descent and Malays are rather underrepresented. This means that the town is an absolute foodie heaven, even more so than the rest of the country. On top of that there’s a lot of really nice architecture, a vibrant arts community, nice cafes and a generally pleasant atmosphere, so the week we spent there passed really really fast. I think at one point in the future I want to spent several month in Georgetown…

Next on our list was going to an island. As I mentioned before our original plans included some Malaysian islands, but if you are already in Georgetown, Ko Tao is quite close and in fact easier to reach. Normally that is, because when we tried to make our way there, there was a train strike in the south of Thailand, as well as a cancelled ferry due to bad weather and several roadblocks for various reasons. This turned our trip into a 24 hour odysee which in retrospect could be funny. Could, but isn’t. This really was one of the hardest and most frustrating journeys of this trip and if I never have to repeat it, it still would be to soon. Anyway, all’s well that ends well, and once we reached the beach, we basically dropped to the activity levels of narcoleptic sloths, put our bodies in basic life support mode and didn’t really bother with questions more complex than “Where should we eat?” for the next ten days. Well almost, because we did sign up for a diving course. I don’t quite know what happened there, but somehow our lazyness (at this point we had been on the road for almost one year already) combined with limited trust in the instructor made us stop after the first day. Maybe I’ll try diving again at a later point, but I’m really not in a hurry when it comes to that.

With our batteries recharged and a couple of days left to spend in Thailand we headed to good old Bangkok, one of my favorite cities in Asia. We stayed at one of our friends’ places, celebrated P.’s birthday with a fantastic dinner, had delicious cakes, saw some of my friends for drinks in the evening and generally had a good time. I always enjoy being in Bangkok, it’s familiar but there’s still always something new to discover.

The next 3.5 weeks we spent in Cambodia, a place I wanted to see for a long time. And it did not disappoint. Sure, Poi Pet is one of the most annoying border crossings in the world and the first impression of Siem Reap is less than favorable, but once you take a deep breath and relax, Cambodia is just wonderful. Or rather, Cambodians are just wonderful. They are open, friendly and funny, often speak good English and enjoy a chat. If you want to know more about the country, there are plenty of opportunities for that! But not only the locals were great, we also met a plethora of nice foreigners, who are really trying to help. But of course it’s no paradise on Earth, so there’s also plenty of shady characters, annoying tuk-tuk drivers, sexpats, opportunists, you name it. Cambodia is a poor country, with a host of problems, and it seems to bring out both the best and worst in people. Don’t let the former blind and the latter deter you and you are bound to have a great time.

Anyway, as mentioned our first stop was Siem Reap, and the Angkor complex (which is way bigger than “just” Angkor Wat) really is as good as you probably believe it is. We got the three day ticket and also explored some of the further away temples, but my personal highlight was returning to Angkor Wat in the early afternoon and having it almost to ourselves for some miraculous reason. We also had a good time exploring the town itself (the “other” side of the river is much more quiet), met some nice expats, went out to party with T. and B. who we met at the border, and had some delicious food. Seriously, Cambodian food is underrated, they have some great stuff! Personally I prefer Thai curries, but a good Amok is almost as good.

Our second stop was sleepy little Battambang, which actually is the second biggest city in the country, but really doesn’t feel like it. Most poeple don’t bother going there at all or just spend a day or so there, but somehow we managed to fill five days without getting overly bored. Of course there’s the standard day trip including the bamboo train, a cool old temple, and an amazing cave where a million bats fly out at sunset, but that wasn’t even that interesting (although we did have a great tuk-tuk driver who explained a lot about Cambodia to us). I really just enjoyed the relaxed vibe of the place, which has a great cafe (Kinyei), an good restaurant if they feel like it (White Rose) and a decent pub (Madison Corner). People are friendly and life has a very slow pace, which I found immensely relaxing.

From there we went to Phnom Penh, a decidely less relaxed place, which nonetheless has its own charms. I don’t like the general backpacker area of town, so I’m glad we decided not to stay there, but instead opted for a guesthouse closer to the BKK1 area before actually moving to another one there. Unfortunately there was a series of attacks on tourists right before we got to PP, but we still walked all over the place to explore it and it felt fine. In fact some areas are really enjoyable and the friendly Cambodians make it even better. We also bumped into two extremely nice Vietnamese guys who were excited to learn that we had just visited their country, and who invited us for some fruit and a chat. Of course we also visited the killing fields and the S21 prison, but I think it’s impossible to truly grasp the horrors that took place in Cambodia. It still was a sombering experience.

Our original plan included exploring the east of the country, but alas heavy rains and floodings prevented us from doing that. So we went straight to lovely little Kampot, a place where we whiled away a week with our new travel buddy, N. from the UK. I’m not sure what it is with me and small towns in Cambodia, but I enjoyed Kampot even more than Battambang. For such a small town there was a surprising lot of good food, they have an excellent and tough pub quiz (which we surprisingly won), and there’s some nice trips you can do from there. We also explored the small town on the other side of the river as well as nearby Kep, and spent quite a bit of time on our guesthouse’s terrace looking at amazing sunsets or at one of the nice cafes reading books. Tough life, I know.

Oops, this got a lot longer than I had originally intended, but I really like all the countries I just wrote about and writing about them brought back some good memories.

The Missing Update

After roughly two months of silence, it’s time for another long overdue update. It covers our stay in Vietnam, or the period from end of July until end of August.

Having spent some time in Taiwan and Hong Kong, we were curious to explore Vietnam, a country that’s supposed to be a bridge between the Chinese world and Southeast Asia. We arrived in Hanoi and I instantly liked what I saw: chaos, street food, old colonial architecture and more street food. Not to forget the bia hoi, delicious (and cheap) beer which is brewed daily. While the traffic is at times insane, Hanoi is a very likable city, featuring some great cafes, excellent food and nice old French houses. Unfortunately a typhoon stopped us from going to Halong Bay, but the Perfume Pagoda made for a nice day trip too.

While many travelers in Vietnam opt for the cheap “Open Bus Ticket” from Hanoi to Saigon with various possible stops in between, we felt like going for a more authentic travel experience and took the night train instead. Alas the sleeper compartments were booked out, so we spent the entire 12 or so hours sitting among Vietnamese families on their way to the beaches further South. Not the best night of my life, but I really enjoyed our destination, the old imperial city of Hue. It’s a lovely town, and while quite touristy, it’s easy enough to get away from it all and enjoy normal Vietnamese life. I also celebrated my birthday there, and the lovely folks of the Moonlight Hotel (yes, we spoiled ourselves there) gave me a super cute Doraemon cake. Thanks guys!

Our next stop was Da Nang, a popular destination for Vietnamese tourists, but apparently not that popular with foreigners. We ended up staying in a suburb close to the beautiful China beach, where English wasn’t really spoken, but people were very curious about us and really helpful. Like many Asians the Vietnamese prefer white skin, so we basically had the beach to ourselves until almost 5pm every day.

We then took the local bus to Hoi An, which is a great little place, that could be so much nicer still if the pretty old houses weren’t all covered in merchandise. It’s still a very nice and likable town though, and we ate some of the best food in all of Vietnam there (check out the Little Menu, it’s a really nice restaurant). For me the best parts where the surroundings though, which we explored on the free bikes our home-stay provided. Cycling through green rice paddies to the beach is among my best memories of our time in the country.

Having made our way back to Da Nang, we embarked on another looong train journey to Nha Trang, Vietnam’s most popular beach destination. We originally had thought about skipping it, but I found a great looking diving school there, and since getting an Open Water license was part of my plans for this trip, we decided to go after all. What can I say, the place is as bad as I imagined, full of obnoxious tourists, construction sites, generic backpacker hangouts and a general air of seediness. Anyway, I got sick, the diving didn’t happen, and while I found that quite annoying, I also was rather relieved to get out of there.

After a relatively short and scenic bus ride we arrived in Da Lat, a pretty little town in a rather picturesque mountain setting. We stayed at Dream Guesthouse, which was a great place and featured one of the nicest breakfasts I ever got in a hostel. They even had Marmite! While the sudden drop in temperature that comes with altitude was slightly annoying at first, the town’s pretty setting and relaxed atmosphere really make up for it. Definitely a place I would go back to!

Since we wanted some more beach time, our next stop was Mui Ne. Getting there was quite a pain in the ass though, as our minibus was filled to the brim with over 30 people and various packages that were dropped off at various destinations on our way, making this dreadful ride even longer. But all is well that ends well, and we had a great time on the beach. We also bumped into some guys we first met in Hoi An and had a fun night out with them and their Austrian (apparently some of us DO travel after all) travel buddies. I also signed up for a kite surfing course, but unfortunately the wind stopped about 30 minutes in and didn’t really make another appearance before we left. Sports just didn’t happen for me in Vietnam… Anyway, Mui Ne was super relaxing, we extended our stay twice and ended up spending almost a week there. I’d almost be tempted to feel silly for that, but we met several other people who did the same.

Our last stop in the country was Saigon, a very different beast from Hanoi. While I seem to prefer the food in Northern Vietnam, I definitely have a thing for the people from the South. They just seem a lot more friendly and open and we had a couple of good random conversations during our stay. We also visited the Cuchi tunnels, but instead of doing it with a boring half-day tour, we did it using public transport, which took about twice as long, but was way more fun (take bus #13 from the terminal and #79 from where it drops you off). I really enjoyed Saigon, it felt a lot more relaxed that Hanoi.

Writing this made me realize just how much I enjoyed Vietnam. Before we went I was curious about the type of experience we’d have, since many people seem to find it a pretty hard country to travel in. However, I found it no harder to deal with than many other places I’ve visited over the years. Sure, some of the sellers are rather pushy, but it doesn’t take too much to get rid of them. And in all fairness, they bother other Vietnamese people even more than they bother tourists. You also often hear that nothing in Vietnam comes at a fixed price and that foreigners often end up paying a premium, but once again I found that they do the same to Vietnamese domestic tourists too. Also I don’t mind paying a couple of cents more every so often, especially when they mean a lot more to the seller than to me. My advice, forget everything you’ve heard about Vietnam before you go there, and approach the country with an open mind and a positive attitude. I’m sure you’ll have a great time!

Oh, in case you wonder, I wrote this in Bagan, Myanmar, but I’m afraid it will take a bit longer before my updates have caught up with my whereabouts…

Great, Greater, Greater China!

By now it’s quite standard that my posts lack behind my travels… This update is no exception, it’s about our travels from end of June to end of July.

The topic: Greater China, or more precisely Shanghai, Taiwan and Hong Kong.

But let’s start from the beginning. In Cali (Colombia) we figured it’s about time to figure out how we’d get from the Americas to Asia. It was pretty clear that we’d have to go through the US, and since A. and V. were not in San Francisco at the dates we were looking at, we decided to visit C. in Los Angeles. After a bit of searching I found a great flight from LAX to Taipei with a 20h layover in Shanghai. P. had never considered going to Taiwan before, but to her credit it didn’t take much convincing and we booked the flight that same evening. Considering that in a past life (or so it seems) I studied Sinology, I did find this quite exciting, after all I studied Taiwan academically but had never set foot in the country. I also dreaded using my rusty Mandarin again.

Not looking forward to spending 20h at the airport, I did a little online search and found out that Beijing and Shanghai now have free 72h transit visas on arrival. Jackpot, especially since I studied in Shanghai and was quite stoked to see the city again. So after a couple of awesome and eventful days in LA (thanks again C., especially for taking us to the Bradbury building of Blade Runner fame!) we boarded our flight and arrived in China 14h later. We were quite jetlagged and tired, but we still bothered to go into the city, where we found a nice little hotel off Nanjing Lu close to the Bund. Before we collapsed we managed to have a nice dinner and a bit of a walk and I remembered why I originally fell in love with this city that I called home for a year. I definitely want to go back for a longer stay at one point…

The next day we arrived in Taipei where we had booked accommodation via AirBnB and a new love affair started almost immediately (yes, I’m a bit of a place whore, deal with it). Over the years I’ve heard many great things about the city, but yet it still somehow managed to exceed my expectations. It’s cool (teenagers have a pretty Japanese style), it’s modern, it’s quirky, and the food is awesome. If you need a break, places like Beitou (very cool public library) or Tamsui are an easy MRT ride away, or you just go to a park and watch old people practice Qi Gong or Tai Chi. Coincidentally one of P.’s friends was in Taipei at the same time as us and we explored the Confucius temple and some other parts together. After a couple of days my friend Ph. arrived from Vienna and kept us company for the next two weeks.

With him we explored more of Taipei, before heading off to Keelong. Which would be transcribed as Jilong in the official Chinese Hanyu Pinyin system, but the Taiwanese seem to use about 5 different systems at the same time, of which at least 2 appear to have been made up solely for the purpose of confusing foreigners. For the rest of this mail I will just go with the transcriptions most commonly found in travel guides, but I digress… Anyway, Keelong was fun. Not the prettiest of towns, but some nice things to see and an awesome night food market. We spent the afternoon with a friendly but slightly insane taxi driver, and at night we found a group of breakdancers to watch. What more can you reasonably expect from a day trip?

The next day we went on to Hualien, where once again we stuffed our faces at a night market. Or several, since we first ended up at the wrong one, where 2 young employees at a food stall were so relieved that they managed to deal with 3 foreigners, that they actually high-fived when we walked away with our food. After we finished our meal they got daring and ventured a “Good food happy?” at us. “my friends, “happy happy!” indeed. The next day was reserved for the famous Taroko Gorge, but a shitty bus service, bad weather and the unfriendliest bus driver of the country made it a rather disappointing experience. You win some, you lose some…

Our next stop was Lukang, where we spent a couple of very pleasant days eating (do you see a pattern here?) and exploring old temples. We then headed down to Tainan, where we basically did the same. More food, more temples. Then maybe a snack. How about another temple? My favorite part of Tainan was Anping, which is definitely worth checking out if you ever make it there.

We now were approaching the end of Ph.’s time with us, and I’m glad he managed to at least get a glimpse of Kaohsiung, one of my favorite new places. I can’t even tell you why exactly, I just really liked it there. Sure, food’s awesome, but that’s basically a given in Taiwan. The people are amazingly friendly, but that’s pretty much standard too. I guess what I like is the overall feel of the place, which is rather relaxed. And artsy. Kaohsiung used to be an industrial town, but most of that industry disappeared. Thanks to a former mayor the old factories and warehouses on the docks now serve as amazingcontemporary art spaces. We came for 3 days, we stayed for a week, andwho knows, without the flight to Vietnam we had already booked at thattime, I might still be there…

Our last stop before returning to Taipei for 3 more awesome days wasTaitung, or rather the countryside around it. Taiwan’s east coast isfar less developed than other parts of the country, but reallybeautiful. Since our hostel had free bikes we managed to explore thearea a bit, alas a second taifun was about to hit (the first onecaught Ph. in Taipei) and so the weather wasn’t exactly great. Atleast we had a great hostel to return to, run by a lovely family,which made us drink way more tea than I thought a human bladder canhold.

If it wasn’t obvious enough yet, Taiwan was one of my favoritedestinations on this trip. The people are just amazingly friendly!Sure, not everybody will speak English, but even then they will stilltry to help you. Or at least smile at you. The food is excellent. Yes,I write about food a lot. Food matters! There’s plenty of old templesaround, actually more than in most parts of China that I visited. Andalongside the old there’s the new. Taiwan has many art and culturespaces, and a vibrant youth culture that borrows liberally from “thewest” as well as from Japan. In a way, our stay there made merediscover my love for the Chinese world, which had died down a bitafter university. I even got back into speaking Mandarin, and while itwas far from perfect, it usually was good enough (or the Taiwanese arejust amazing at reading body language in combination with nonsensicalphrases.) If you are looking for an intro to the Chinese culturalsphere that’s easier than the mainland, I can wholeheartedly recommend Taiwan!

After the weeks of relative tranquillity, Hong Kong (one of myfavorite places ever) was quite the change. It’s louder, busier, andmore stressful. On a superficial look it’s all modern and businesslike, but if you dig deeper, you’ll still find bits and pieces of oldworking-class Hong Kong. And if you want a break, you can always justflee up Hong Kong hill or head out to Lantau to visit the peacefulfishing village of Tai O. We stayed in Chungking Mansions(http://en.wikipedia.org/wiki/Chungking_Mansions), which despite therecent renovations is still a rather unique place. After all thisplace had a book written about it, the aptly named “Ghetto at theCenter of the World”. Another nerd dream came true when we visited theonly Studio Ghibli store outside Japan, where we took pictures withTotoro and in the Catbus. Yes, I was inside the Catbus, how cool isthat? Unfortunately the weather was really bad during most of ourstay, so P. didn’t get to fully enjoy HK, which is a bit of a shame.Oh well, guess that means I will have to take her back at one point…

Photos I hear you say? Here you go:
Taiwan
Hong Kong

Confucius say “Be good and travel more!”

Extending Nmap With Lua

I originally wrote this piece for a pentesting magazine, but after some unprofessional behavior by their editor I lost my cool and decided not too publish it there. It’d be a shame to let it go to waste though, so I after almost three month I finally decided to put it up on my blog.

Whether you are working as a security professional or a network administrator, chances are that Nmap (“Network Mapper”) is part of your regular toolkit. For many people this project, which was started by Gordon “Fyodor” Lyon in 1997, is the first choice when it comes to host and service discovery. But thanks to the Nmap Scripting Engine (NSE), it can do much more than that, allowing users to easily develop and share their own scripts, thus turning it into a versatile security scanner.

What you will learn…

  • basics of the Lua programming language,
  • walkthrough of a simple NSE script,
  • where to find more information on scripting Nmap and Lua in general.

What you should know…

  • basic Nmap usage,
  • an imperative programming language.

The Nmap Scripting Engine (NSE)

Scripting has been part of Nmap for a while, but version 6 includes a dramatically improved implementation of the NSE infrastructure. This also led to an enormous increase in available scripts, from 59 in version 5, to 347 in the latest release and more than 430 in the current SVN repository. The core of NSE is an embedded Lua interpreter. Before NSE’s implementation started, the Nmap developers considered various programming languages for inclusion, but finally settled on Lua because it’s small, liberally licensed (MIT), fast, actively developed and designed with embeddability in mind. It’s also used in other popular FOSS security tools like Wireshark and Snort, as well as in popular projects such as World of Warcraft or the window manager Awesome.

The Lua Programming Language

Development of Lua started in 1993 at the Pontifical Catholic University of Rio de Janeiro, Brazil. It was designed by Roberto Ierusalimschy, Waldemar Celes and Luiz Henrique de Figueiredo, with the primary goals of being lightweight, cross-platform and easy to embed in other projects. These goals were achieved by writing the language in portable ISO C and giving it a comparatively simple C API. Also Lua’s interpreter only takes up about 180 kB in compiled form, so embedding it does not incur a huge cost. Over the years it became a popular scripting and extension language, used in a wide variety of projects. The syntax of Lua should be familiar to anyone who has experience in Modula or C style languages (control structures like if, while, for), but the designers borrowed liberally from other languages as well: the semantics of Lua are heavily influenced by Scheme, and even though the different syntax obscures this a bit, it becomes rather clear the longer you use the language. In some ways it feels like a cleaner and better designed JavaScript, so it’s normally easy to understand a given snippet of Lua code even with little or no previous exposure to the language.

Key features

Lua features first-class functions, thus allowing the developer to treat functions like any other data type. It also doesn’t impose a certain programming paradigm, but instead provides meta-features to extend the language as needed. This for example makes it possible to implement things like inheritance in a rather simple and straightforward way without explicit language support. Lua is also garbage collected, features closures and proper tails calls and provides an infrastructure for cooperative multitasking in the form of coroutines. While most of this goes beyond the scope of this article (and NSE), interested readers can find more information in the excellent book “Programming in Lua”, an older version of which is available for free on the project’s web site.

Tables

There’s is however one more feature of Lua that’s worth discussing in a bit more detail, namely its reliance on a single compound data structure, the table. Tables are in essence what other languages commonly refer to as associative arrays or dictionaries, or key-value pairs. This is how you create a simple table:

1
table = { x = 10, y = "foo" }

The values can be accessed like this: t[“x”] and t[“y”]. For string keys Lua supports a nice shorthand though, so t.x and t.y will produce the same result.

One special feature of Lua’s tables is that they are also optimized for use as arrays, which can be created like this:

1
array = { "citizen", "428", "blog" }

Indices are assigned automatically and unlike in other languages start from 1, not 0 (although it is possible to explicitly assign to array[0]). So to get the string “428” out of our array, we have to use array[2].

With this comparatively simple data structure (and the powerful metatable feature) Lua not only implements arrays and dictionaries, but also records/structs, namespaces, classes, objects and much more.

Examples

Enough theory, let’s look at some example Lua code. First the customary “Hello world” program:

1
print Hello world

This will just output Hello world. Now for a function definition:

1
2
3
function sayHello(name)
  print ("Hello " .. name .. "!")
end

To generate the same output as above, we would have to call the function with the appropriate argument: sayHello(“world”). Alternatively, the same function could be defined like this:

1
2
3
sayHello = function(name)
  print ("Hello " .. name .. "!")
end

Last but not least a small loop that iterates over an array and prints out each element (ipairs returns the index-value pairs of an array):

1
2
3
for i, str in ipairs(array) do
  print (i..": "..str)
end

Assuming the variable array contains the elements “citizen”, “428”, and “blog”, this will generate the following output:

1: citizen
2: 428
3: blog

NSE example script

The Nmap web site contains some excellent documentation regarding NSE (short intro, full book chapter, tutorial, so instead of repeating all of what’s written there, we’ll look at a practical example instead. The script shown here is http_generator.nse, which looks for the presence of an HTTP generator meta tag to determine which CMS the scanned hosts run (if any). This information can be useful, since outdated versions of such systems are common attack vectors.

Library imports

1
2
3
4
local http = require "http"
local shortport = require "shortport"
local stdnse = require "stdnse"
local string = require "string"

It’s customary to start Nmap scripts by importing all the needed libraries. In the process they are assigned to local variables, so if you wanted to access them under a different (presumably shorter) name you could do so (for example local s = require “string”). There’s a wide variety of available libraries, from protocols (afp, dns, http, bitcoin, openssl etc.) over encoding/decoding (base32, base64, json) and utility libraries (nmap, packet, pcre) to a framework for brute-force attacks against services (brute). The script we are looking at uses the http library for communication, shortport for an easier way to define portrules (more on that later), and stdnse, a collection of utility functions provided by NSE. The string library is a standard Lua library and used for string matching in this example.

Metadata

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
description = [[ Displays the contents of the "generator" meta tag of a web page(default: /) if there is one.
]]

author = "Michael Kohl"
license = "Same as Nmap--See http://nmap.org/book/man-legal.html"
categories = {"default", "discovery", "safe"}

---
-- @usage
-- nmap --script http-generator [--script-args http-generator.path=<path>,http-generator.redirects=<number>,...] <host>
--
-- @output
-- PORT    STATE SERVICE
-- 80/tcp  open  http
-- |_http-generator: TYPO3 4.2 CMS
-- 443/tcp open  https
-- |_http-generator: TYPO3 4.2 CMS
--
-- @args http-generator.path Specify the path you want to check for a generator meta tag (default to '/').
-- @args http-generator.redirects Specify the maximum number of redirects to follow (defaults to 3).

-- Changelog:
-- 2011-12-23 Michael Kohl <citizen428@gmail.com>:
--   + Initial version
...

After the library imports, scripts contain some metadata that is used for documentation and internal purposes. The description and author variables are self-explanatory, the syntax used for the former is Lua’s way of creating so-called “heredocs”, string literals that preserve white space and line breaks. In regard to the license script authors have two choices, the GPL-like Nmap license, as well as a simplified (2-clause) BSD license. Last but not least there’s documentation in the NSEDoc format. It’s customary to include usage documentation, example output, descriptions of all the available arguments (if any) as well as a changelog.

There’s another important field that this specific script doesn’t use, dependencies. This defines the order in which scripts should be run and is used for the case where one script works on the results of another script. It’s important to keep in mind that scripts listed here will not be automatically run, they still have to be selected via —script scriptname or in another way (for example —script=default).

The categories variable is a bit more interesting, it defines which groups of scripts our current program belongs to. Scripts marked as default will be run automatically when Nmap is started with the -sC or -A options.To qualify for inclusion in this category, scripts have to adhere to some criteria: they need to be fast, reliable, non-obstrusive and generally useful. There are no clear definitions for these criteria, it’s up to the Nmap developers to add a script to this category. The discovery category collects all scripts that actively try to gather more information about the scan targets, in our case the “generator” HTML meta tag. If a script is marked as safe, that means that it was not written with the intent of crashing services or exploiting vulnerabilities and that it’s unlikely to consume large amounts of resources. There are many more categories Nmap uses, for example auth (scripts dealing with authentication credentials), brute (tools for brute force attacks) or vuln (checking for specific vulnerabilities). For a full list, please consult Nmap’s excellent documentation.

Now that we know a bit more about categories, let’s see how we can use that knowledge for more interesting Nmap runs. Let’s start by running all the default scripts:

1
nmap -sC example.com

or

1
nmap --script=default example.com

The invocation nmap -A example.com (“Aggressive/advanced scan options”) enables OS and version detection, script scanning (equivalent to nmap -sC), and traceroute.

If you just want to run a single script, you can do so by specifying the —script argument, for example nmap —script http-generator example.com.

It’s also possible to run scripts from several categories, by using something like nmap —script “default,safe” example.com or nmap —script “default or safe” example.com. There is an analogous and operator, so nmap —script “default AND safe” example.com will only run scripts that are in all the listed categories.

As an example for a rather complex invocation, the following command will run all scripts from the three specified categories, except for the ones whose name starts with “http-”.

1
nmap --script "(default or safe or intrusive) and not http-*" example.com

Last but not least it’s also possible to run user scripts which are not part of the standard Nmap distribution, as show in the following example:

1
nmap --script default,banner,/home/user/customscripts example.com

The above invocation will run all default scripts, the script named “banner” and all scripts which can be found in the directory /home/user/customscripts.

Rule

The rule definition is an important part of every NSE script, it determines when the script should run. A rule is a Lua function that returns true or false to decide whether or not the script’s action function will be executed. There are several different rule types, depending on when the script is supposed to run (before or after hosts are scanned, prerule and postrule) or if the trigger is a specific host or port (hostrule and portrule).

The rule definition in our example script is very simple since it uses the very handy shortport library, which has several predefined rules for common scenarios:

1
rule = shortport.http

This will match and return true when a given port is likely to be an HTTP port.

Action

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
action = function(host, port)
    local path = stdnse.get_script_args('http-generator.path') or '/'
    local redirects = tonumber(stdnse.get_script_args('http-generator.redirects')) or 3

    -- Worst case: <meta name=Generator content="Microsoft Word 11">
    local pattern = '<meta name="?generator"? content="([^\"]*)" ?/?>'

    -- make pattern case-insensitive
    pattern = pattern:gsub("%a", function (c)
                  return string.format("[%s%s]", c, string.upper(c))
              end)

    local response = follow_redirects(host, port, path, redirects)
    if ( response and response.body ) then
        return response.body:match(pattern)
    end
end

The action function is the heart of every NSE script, it contains the code that will be performed when the rule matches. Basically action is just another Lua function that accepts the same arguments as the rule function (which is not as obvious in our example because of the use of the shortport.http rule).

The function starts with checking for the presence of two arguments and specifies defaults in case the user didn’t provide any values for them. Arguments can be passed to the script in the following way:

1
nmap --script http-generator --script-args http-generator.path=/cms,http-generator.redirects=2 example.com

They get stored in the table nmap.registry.args, but instead of using them directly you should use the function stdnse.get_script_args to access them.

After the argument handling, a pattern matching the HTTP “meta” tag is specified. Standard Lua does not include a regular expression library, but uses its own patterns instead. While there is a library providing Perl-compatible regular expressions (PCRE) for Nmap, it seemed like overkill to drag it in for a single expression, so I relied on plain Lua for this, which explains the little workaround used in the next line to make the pattern case insensitive by replacing every letter (the pattern %a) with its lower and upper case version. After that the follow_redirects helper function is used to get an HTTP response, and if there is one, its body will be scanned for our pattern.

If there is a match, the function will return the matching string and Nmap will automatically include it in its output as shown here:

PORT   STATE SERVICE
80/tcp open  http
| _http-generator: WordPress.com |

Instead of a string the action can also return tables consisting of name-value pairs which are then formatted in a structured fashion, whereas the return value nil generates no output at all.

As you may have noticed I didn’t yet show you the helper function mentioned above, so while it’s not very interesting in itself, I’ll include it for completeness’ sake:

1
2
3
4
5
6
7
8
9
10
11
local follow_redirects = function(host, port, path, n)
    local loc
    local pattern = "^[hH][tT][tT][pP]/1.[01] 30[12]"
    local response = http.get(host, port, path)
    while (response['status-line'] or ""):match(pattern) and n > 0 do
        n = n - 1
        loc = response.header['location']
        response = http.get_url(loc)
    end
    return response
end

This helper function is a simple way to follow HTTP redirects. It first defines a pattern for recognizing the relevant HTTP status codes (301 and 302) before making a request to the specified host, port and path combination. It then checks for the presence of a redirect and if there is one, follows it to retrieve the location information from the new location. When there are no more redirects or the redirect limit (specified in the argument n) has been reached, the function returns the full HTTP response to the caller.

Digging deeper

And that’s all there is to it! Thanks to the excellent infrastructure provided by NSE and all its libraries, writing custom Nmap scripts is a pretty easy and straightforward task. Anybody with a bit of scripting experience should be able to pick up the basics in a very short time and then learn about the finer points while developing scripts. The Nmap development team has assembled some excellent documentation which can be found online at http://nmap.org/book/nse.html. There’s also a great presentation by Fyodor and Nmap co-maintainer David Fifield available online which gets you started on developing your own NSE scripts in less than an hour.

Summary

Nmap is a popular tool for security and network professionals around the world, and its excellent scripting capabilities make it very versatile. The Nmap Scripting Engine relies on the popular and easy to learn Lua programming language, and provides useful libraries for quickly and efficiently developing your own custom scripts, which can then be shared with the wider Nmap user community.

¡Viva Mexico!

Oh man, this is one overdue post. I wanted to write about Mexico forever, but then we were traveling in Taiwan, and suddenly the whole Latin America experience just seemed so far away already… Anyway, we are now in Vietnam (Hanoi to be more specific), so I thought I’d finally sit down and write up my impressions.

Mexico was one of the places I wanted to visit for a long time, so before we went, I was a bit nervous that it may not live up to my expectations. But not only did it do that, it actually far exceeded them! There are fantastic things to see, the food is exceptionally good (also for vegetarians) and the people amazingly friendly. Also it felt like a really safe and easy place to travel: as opposed to big parts of South America the security situation in Mexico is pretty relaxed, even though stuff does of course happen. On top of that there are good roads everywhere and ADO is a very nice bus company, so we generally didn’t dread moving from A to B. With that said it’s quite unfortunate we only had 4 weeks there, but we had booked the flight to Taiwan beforehand in order to avoid getting stuck in the Americas for all of our trip. Turns out that was a good idea, because Mexico alone could easily have kept me entertained for several more month.

Our trip started in Tulum, which features a beautiful beach and a cute little set of ruins. Unfortunately Chaak (the Mayan rain god) wasn’t on our side, so we only managed to get one and a half proper beach days. On the upside we got to taste tons of good food and also explore the nearby ruins of Coba. We then went to lovely Valladolid and the ruins of Ek Balam, both of which are a bit overlooked by tourists (or just crammed in on day tours). I’m glad we stayed there, a super relaxed little town, and the ruins were among my favorites in Mexico. Next stop was Merida, which is a decent place in itself, but also the gateway to the ruins of Uxmal. A decent place, but slightly tainted by the memory of annoyingly obnoxious hostel neighbors. If you’ve been to Mexico before and wonder, yes, we did in fact skip Chichen Itza. Based on the feedback of people we met, it really did not seem worth the effort in time and money. Instead we decided to give another little and often overlooked town a chance, and I’m definitely happy that we went to Campeche. Quiet, with a small but pretty old town and very down to earth, this is the sort of place I really came to enjoy on this journey, and not only for the lack of a specific type of backpacker I detest. After this brief stop we went on to Palenque, a set of ruins I had heard lots about, and which I found to be a bit disappointing. Letting vendors onto the premises really doesn’t add to the atmosphere, even if they don’t hassle you… At least the setting of our hostel was very scenic, in that small cluster of hostels right at the beginning of the national park. The next stop was one of my absolute favorites, San Cristobal de las Casas where we ended up staying almost twice as long as we originally planned. True, it’s small, but thanks to several cultural centers there’s a surprising lot to do in town. It’s also Zapatista heartland, so if you have any interest in the EZLN, this is THE place to be. Our trip to the coast around Puerto Escondido got canceled due to a hurricane, so we went straight to Oaxaca instead, which was also quite nice. Unfortunately P. got sick, so I mostly explored it by myself including a Mezcal factory, a petrified waterfall, and of course more ruins. Last but certainly not least was the D.F. (Distrito Federal), more commonly know as Mexico City outside of the country. You have to understand that I wanted to visit this city since my elementary school days, when I first heard that it’s the biggest in the world. It may not be anymore, but it’s still one of the most interesting places I ever visited. It’s hard to put the city into words, but it has everything: the old, the new, the very poor, the super rich. It’s a giant, it’s a moloch, and yet, it’s lovely. While not all of it is pretty, the parts that are, really are. When you walk into Coyoacan or San Angel for the first time, it’s hard to believe that they are part of Mexico City at all. And given the size of the place, it’s incredibly how friendly and helpful most of the people still are. BTW, thanks again for dinner and a great chat Hector! It’s hard to put Mexico City into words, you better go there yourself!

No post about Mexico could be complete without a separate mention of the food. I always considered Mexican cuisine to be one of my favorites, but I wasn’t sure how authentic it was. Turns out that the things we know like tacos, burritos, quesadillas, enchilladas, fajitas, and guacamole traveled rather well to Europe, although unfortunately we can’t find them at tasty little street stalls where they are almost ridiculously cheap at times. But there’s just so much more: sope, gorditas, molletes, tortas, papadzules, brazo de reina, tlayudas, and a ton of delicious things I can’t remember the name of. Just writing about it now makes we want to go back right away…

Mexico definitely was one of the highlights of this trip for me (actually of all my travels so far), I’d instantly go back.