Weird quirks of IntelliJ IDEA: GUI Editor Rendering Problems

So, I’ve been building a little analysis software that involves building a barchart. I simply do that by drawing out some rectangles in a JPanel according to the size of the window and the size of the bars. After a late night coding session, I tried reopening IntelliJ and wanted to add a few new buttons to the GUI. But I was greeted with a really glitchy interface.

Well, turns out: When the code behind a custom view part throws a NullpointerException, the GUI editor somehow has a hard time recovering from that and rendering a preview.

Fixing your code also fixes rendering issues in the IntelliJ GUI editor.

Author’s notice: For reasons unbeknownst to me (magic intervention of the IntelliJ team?), I couldn’t reproduce this error properly. Sometimes it would happen, sometimes it wouldn’t. I suspect it also had something to do with the internal caches that IntelliJ uses, but that is pure speculation.

How does WhatsApp Web for iOS work?

A good while ago, WhatsApp developers decided to roll out a feature called WhatsApp for Web on Android devices. Essentially, messages that your phone received would get mirrored on your screen, and you could use your comfy desktop keyboard instead of using those nasty virtual keys to answer all the millions of messages you get every day. For eight months (if my sources are correct), this wonderful feature for all of us who know touch-typing and prefer big screens over tiny handhelds was only available for those who were blessed with the presence of an Android device – iOS users were left out. And rightfully so.

Why? What’s the problem?

You see, on iOS, multitasking works a bit different than on Android. To quote the official Apple documentation:

When the user is not actively using your app, the system moves it to the background state. For many apps, the background state is just […] the app being suspended. The app is in the background but is not executing code. The system moves apps to this state automatically and does not notify them before doing so. While suspended, an app remains in memory but does not execute any code.

It also clearly states the only exceptions to this rule, is using one of the following background modes:

  • Apps that play audible content to the user while in the background, such as a music player app
  • Apps that record audio content while in the background
  • Apps that keep users informed of their location at all times, such as a navigation app
  • Apps that support Voice over Internet Protocol (VoIP)
  • Apps that need to download and process new content regularly
  • Apps that receive regular updates from external accessories

I guess the closest thing that WhatsApp Web is trying to accomplish would fall under either VoIP or downloading and processing new content regularly. But there’s something that makes both of these unusable: The problem with the VoIP background mode is that Apple is very harsh with rejecting apps that try to get into the app store while (ab)using APIs for any other than their intended purpose. See Guideline 2.16 of the Apple App Store Review Guidelines:

  • Multitasking Apps may only use background services for their intended purposes: VoIP, audio playback, location, task completion, local notifications, etc.

 

Well, you say, they could just use the excuse that they are an “app that needs to download and process new content regularly” – and you’re not wrong. But looking at the documentation for “Fetching Small Amounts of Content Opportunistically” reveals why exactly this is not the desired background mode to use:

Apps that need to check for new content periodically can ask the system to wake them up so that they can initiate a fetch operation […]. Enabling this mode is not a guarantee that the system will give your app any time to perform background fetches. The system must balance your app’s need to fetch content with the needs of other apps and the system itself. After assessing that information, the system gives time to apps when there are good opportunities to do so.

In other words: if iOS feels like it, then, maybe your app will get some background time to work with. But that’s not what we want when we are doing some messaging, messages flying by second after second, images being sent and all. No, what we want is a consistent way to get the messages across. And that means also scratching this idea as a way of achieving our goal.

So how would you do it if not in the background?

Well, a few reddit threads and very non-technical and even less satisfying news articles later, I decided to conduct my own research. Here’s some behaviour that can be seen when using the WhatsApp Web Version with iOS:

  • When the phone is connected to the internet and paired with the Web Version, all messages that appear on the phone also appear on the computer.
  • When messages appear on the computer, the phone screen turns on and the vibration can be felt, but the notification vanishes from the lock screen (as you have already read the message on the computer).
  • If you put the phone into airplane mode, WhatsApp Web complains that it can’t receive any more messages.

Looking at our observations, we can make a few assessments with fairly high certainty:

  • The phone is playing a critical role in the process of receiving and/or sending messages using WhatsApp Web
  • The Web version has to have some kind of connection to both the WhatsApp servers and the phone, as it can assess when the phone is not available over network.

So I dug deeper into the documentation of background modes in iOS, with no avail, I tried my best to remember whether I had heard anything during my time learning using CS193P, and I finally figured out the solution.

Using iOS Notifications to accomplish the cause

I almost didn’t remember, but there is another almost-background mode that iOS allows to be used. “Using Push Notifications to Initiate a Download“. Combining the previous knowledge with reading a fair bit of the documentation, here’s how it’s done:

  • On a paired phone, when a new message arrives, the push notification on the phone gets shown.
  • Thanks to Apple’s Push Notification API, the WhatsApp application gets called in the background (behind the lockscreen) and is allowed some processor time.
  • It uses the processor time to make a call to a WhatsApp API, probably checking whether there is a client available, and relaying the message that has arrived on the user’s phone back to the web version.

Now, you would think, sending messages should be easy, just push it right from the web interface to the WhatsApp servers and boom, you’re done! To be honest, I think the same, but for some reason (safety maybe, or other technical reasons), here’s what actually happens:

  • WhatsApp Web uses the Apple Push Notification Service (APNS) to send an invisible notification to the iPhone with the message to be sent as the payload.
  • This causes the app in the background to be woken up once more, do its job and send the message out into the world.

Whilst a bit complicated and very tricky indeed (especially compared to the Android version), it’s a nice and honestly quite battery-efficient way of relaying the messages via mobile phone (since it works mostly over APNS, and does not require for a third-party server to be constantly running on the phone). I had some great fun dissecting this topic. It’s always great to see how people get creative when they run into limitations.

Disclaimer: Due to the closed source nature of WhatsApp, some of the details (e.g. how exactly a payload is structured, or what is sent at which point) might be a bit off in this article. If you have any more information or feel that something stated in this article is wrong and would like to share it, feel free to drop me a line.

Quick Trick: Using an SSH enabled machine to quickly build a SOCKS proxy.

Let’s just jump right into the scenario: You are located in an European Country and want to access some of those new hip streaming websites. Now, you need to circumvent the barrier set in place… And how do we do that? Exactly, we build a quick and dirty proxy server, that you can use in of your home network! Let’s get started. For this recipe, you’ll need:

  • An ssh-enabled thing (server, vserver, jail, just anything with a commandline and ssh access)
  • A Raspberry Pi or something else that’s linux based and that you’re comfortable with running longer times (though that is not a necessity)
  • A web browser that’s capable of using proxies (I think that’s literally every single web browser out there)

Now, let’s get started here, okay? This is going to be a fast one. On your Raspberry Pi (or whatever local Linux machine you wish to use), do the following:

sudo apt-get -y install autossh
autossh -f -M 20000 -D 0.0.0.0:7070 root@RE.MOT.EIP -N

Boom! Now, I can’t just let that be like this. Let’s hop in with a couple of explanations! I’ll steal a bit from the manpages here.

We install autossh via apt-get, which should still be pretty straightforward and work on all systems using apt as the package manager. The -y flag just skips all prompts for convenience. But why autossh? There’s a simple reason for this: If your ssh connection fails (e.g. because you’re in over wifi or you trip over the cable), it’s a lot easier to have the service reconnect on its own. And trust me, this version is very reliable. In fact, I have yet to experience a single outage that required a manual restart of the service.

-M 20000 specifies the base monitoring port to use. Without the echo port, this port and the port immediately above it ( port + 1) should be something nothing else is using. autossh will send test data on the base monitoring port, and receive it back on the port above. For example, if you specify “-M -20000”, autossh will set up for‐ wards so that it can send data on port 20000 and receive it back on 20001.

-f causes autossh to drop to the background before running ssh. Which is exactly what we want, because this way we can detach from the session without problems.

-D is not an autossh command, but is forwarded to ssh. It specifies a port forwarding in form of a SOCKS5 proxy. We bind to 0.0.0.0, which means that the proxy is available for all IPs trying to access, not only localhost. This way, your whole home network can access the SOCKS-proxy. If you’re concerned about security, you can edit the binding to your liking.

root@RE.MOT.EIP just tells the ssh instance with which user to connect to which server.

-N Do not execute a remote command. You do not interface with the server at all, you just want the sweet sweet port forwarding.

Now, if you’re one of the cool kids, you can throw the line mentioned above into your crontab (by using crontab -e) and have it set to @reboot:

@reboot autossh -f -M 20000 -D 0.0.0.0:7070 root@RE.MOT.EIP -N

As simple as that. Now, since you do not want to enter your password every single time, it’d be advised you upload your SSH key to the server you are connecting to. I’m not going to explain you how to do that in this post, though, there’s enough tutorials out there.

Have a good one, and good luck streaming! 😀 If there’s any questions, feel free to ask in the comments. That’s what they are they for. Otherwise the Spam Bots might get lonely. 😛

CF.lumen: The real f.lux for Android

If you work on your computer or phone at night times, sometimes you get the feeling like staring at the screen too much is straining your eyes. Now, whilst there are different people arguing whether or not it helps, I certainly find it very nice to have a so called “temperature filter” on my screen: That means that the spectrum displayed is different during the daytime (in which it is more alike with the spectrum of natural sunlight) compared to nighttime (in which it resembles more the spectrum of a candle). For this, I use the great and free software f.lux, available for Windows, Mac OSX and iOS. Notice how this list misses the obvious – Android? Well, the official f.lux website only says:

When is the Android verson coming out?
We have a version internally (it looks beautiful!) but it requires a very complicated installation process. We are working to simplify this and ship f.lux to the Android OS as soon as possible.

In other words… there is none just yet. (At least it’s not publicly available, which is a shame). So, what does one do? One seraches. Let’s see what a Google Play Store search brings up for “f.lux”. One very promising example is “Lux Lite“. Almost sounds like the real deal, doesn’t it? But upon further inspection, it reveals as something like this:

luxlite

Now, that’s all good and nice, and seems like it would be a good way to change our screen colour. But wait a second… Could it be that someone just used a half transparent red rectangle and played around with the transparency setting a bit too much? That’s not even close to what we want! This distorts contrast and visibility, actually quite a lot even! With f.lux, you get such a nice and subtle effect, whilst this is rather blatant. So, back to Google.

After a few more or less successful moments of searching around and browsing the XDA forums, I got to a project called Chainfire 3D. Together with some plugins, you would be able to scramble something up that’s suitable for whatever we wanted to achieve in the first place. But, Chainfire 3D is no longer officially available for devices running JB and so on. But, apparently there have been a lot of requests to the developer to at least get the f.lux like functionality ported to KitKat and co, this is the reason I can link you today to:

Chainfire’s [root] CF.lumen (free) on Google Play

If your phone is rooted, you can simply click the link above and install the app right away, for free. If your phone isn’t rooted, you might want to look into that topic. I know that it might sound a bit scary at first, but liberating your device is one of the few steps that can really help you unleash the full potential of your personal mobile phone.

CF.lumen is what you’ve been searching for all along. It’s essentially a perfect f.lux implementation under Android, available completely for free. However, there is a pro version as well that you can get for a few bucks. The developer, Chainfire, is extremely active in the Android community and has brought all you people important apps that many of us couldn’t use their phone without anymore, for example SuperSU or TriangleAway. Also be sure to check out his latest addition, Pry-Fi, in case you are concerned about your privacy. So please, consider throwing a few dimes in his direction so he will continue the great work he does and will support the apps he provides.

Have a great time, and enjoy all your stuff at nighttimes without having an “eerie blue glow” at night. :)

Visiting Files in the Home Network

Now we’ve all been in a situation where we wished to be able to access something in our home network. Some of us even own a home server, and even fewer people actually set up the home server web interface (speaking of Windows Home Servers, that is). It’s a bit difficult to do, and in my case, I for some reason couldn’t be bothered to reconfigure it after I had to exchange my home server on warranty. So, now, I was finding myself at another place in time and space, and was left with no real way to access that file I needed so badly over the air. So, what do you do in such a situation? You get inventive.

The Requirements.

For my next trick, I will require:

  • A functioning internet connection at home, and an already configured DynDNS service so you can actually access your home network
  • A home server (that’s currently on)
  • A raspberry pi or other Linux-powered device (that’s currently on)
  • A shell (no, not the one you find on the beach. The Linux one)
  • A good bit of time and imagination

Let’s get started, shall we?

Our goal.

Now we could do some magic with sshfs, which would be the proper way. But imagine you find yourself at a friend’s house and want to access, let’s say, last year’s holiday videos. You want a quick and dirty way to access the files that are not directly available on the internet. So, we shall take a different route: We will connect to the RasPi (or Linux Box), mount the Windows Home Server CIFS share, and reroute it into our local webserver’s file. Once all of that is done, it should be easy as 3.1415 to actually access those files.

Establishing the connection.

Our tutorial requires a bit of luck. Or, you could also say, a bit of precaution. Being the good programmy person that you are, you of course already set up your home router to forward your ssh ports to your linux device of choice, be it a RasPi, a BananaPi, or whatever other products with the name of pastry might be out there that resemble a linux box. Typically, that would be port 22. So, how about you do a simple

ssh user@my-home-network-addr.net

and see if this works? It does? Great. That means the biggest hurdle is already overcome. From hereon, it’s a pretty straightforward process.

Naturally, you would have already installed a lightweight server distribution on your raspberry pi by now that you can access over my-home-network-addr.net (or, maybe even better, with HTTPS). Before we work our way through the actual mounting, let’s create a mountpoint. Ideally, you’d put that somewhere where you can remember. So mkdir a folder of your choice in an accessible location. So, now, we can just go ahead and do some mount magic. Please sudo up that nano and add the following line to your /etc/fstab file, and alter as needed.

//IP.OF.YOUR.HOME.SERVER/Path/To/Desired/Folders /your/mountpoint/goes/here cifs username=Your_Username_Goes_Here,password=Guess_What_Goes_Here,workgroup=your_workgroup,users,auto,user_xattr 0 0

Now, save all of this and execute that sweet sweet sudo mount -a. Now, if you cd yourself into the mountpoint of your choice, you should already see all the desired files. Smells like victory already, doesn’t it? That’s right. Now, only add a ln -s the desired parts of your mountpoint into your /var/www folder to make sure it’s accessible over HTTP. At this point, if you have directory listing enabled, you should already be good to go, and your files should be available to download via HTTP on any PC you’d like to try it.

As a bonus exercise, do all of this on Android via some SSH implementation to show a friend running Windows a couple of photos. 😛

I might do a couple of tutorials about how to install the requirements for your server (DynDNS, Lighttpd, MySQL, PHP, and so), but I’m not going to lie, that will heavily reply on my daily feeling. So, stay tuned, and feel good with all the stuff you accomplished today!

A Social Network for None of us.

We live in a saturated world when it comes to the media. Of course, you could argue, we haven’t reached the peak of our potential just yet. We’re just at the beginning of new technologies like wearables, for example Google Glass or Android Wear in general. Of course, that’s true. But that doesn’t change the fact that the way we interact is on a level where we probably would’ve never thought it. In this article, I’m going to highlight a few of the changes our everyday communication has lived through in the recent decade or two.

If you don’t smile, you frown

Ever since Emojis have been introduced to the iPhone world, the way we communicate has drastically changed. People were suddenly using those “Smilies” even more than they used the text based ones just a few years ago. What first brought up some confusion when being exchanged between iOS and Android (back in the good ol’ days where Android couldn’t show Emojis correctly, or at least extremely ugly) has now become a means of everyday communication. I know enough people whose conversations on iMessage, WhatsApp and the like more or less only consist of a single word and half a dozen Emojis. Now that’s nice and all, who doesn’t like a few little pictures to illustrate the said words. I personally also use Emojis, a lot, but I also know a good portion of people who refuse to use them at all. And that’s very interesting. The better with words people are, the lesser they depend on emojis to show emotions, to show what they mean, to clarify sarcasm or irony. Emojis simplify, and the lack of emojis can often be mistaken as grumpiness or being overly serious.

k.

We live in a world with absolutely magnificent data transfer rates, we communicate using our phone, our laptop, our PC, and so on. Yet people somehow can’t figure out it might be taken as rude if one’s reply only consists of a single letter, maybe an exclamation point after it. The lack of words in this world is truly frightening. I personally find rhetorics and the like highly interesting, entertaining and useful. The lack of such however produces two-letter-messages that in my personal opinion especially emphasise lack of respect for the conversational partner. Unless one is driving, or there is something that is directly prohibiting you from texting, then please, use your single letter answer, or better yet, don’t text at all. But in all other situations, I think everyone appreciates to take literally two seconds out of your day and reply with “Okay”, “Alright”, or something similar.

tl;dr

The last “problem” that I would like to address before concluding is the attention span nowadays. We all know that we live in a very rapidly moving society, and that it is hard to keep up with all the news, the exciting things happening in technology, politics, design, music and so on. A natural, yet very unpleasant phenomenon is a very short attention span. This also explains so called “Random” YouTube videos. Quickly cut, mostly just scenes seen as “funny” thrown together, compiled in an under-two-minute-video so god forbid you would have to concentrate on one thing for too long. I don’t blame anyone, I sometimes have a hard time focusing on a single thing as well and tend to multitask (though I still get around to stopping one task if I can sense that it inhibits the other from unfolding its full potential). The real problem is that this is a very hidden development, most people do not realise, and especially the younger generation seems to have a big problem with that. If I see people getting nervous because they don’t have their iPhone and can’t feed their brains on the newest Instagram feed items, I develop the feeling that this tendency is a very unwanted trait. I do not blame people on using their time on something else than I do, it is in no way a “waste” of time if they truly enjoy what they are doing, if they get happiness out of that. There might be people who are really happy when they see new pictures by amateur photographers, when they can see what kind of reviews people post under their entries and so on. But I have the feeling like the majority of these participants are just there because they are bored by whatever is happening around them, simply because the world surrounding them can’t feed their brains with flashy colorful pictures from all over the world at all times. So instead of enjoying the weather, people chose to stay out of touch with reality, and rather entertain themselves with something completely artificial and fictional – not because they enjoy it, but because actually trying to have a conversation with someone in the room requires a tiny amount of effort, namely saying “Hi”, maybe standing up beforehand in case the desired conversational partner is not in direct reach.

Not a solution. Not even an experiment.

There certainly is no overall solution to this new trend, and quite frankly I don’t think the world as it is needs changing. But, wouldn’t it be funny to imagine something that counters those tendencies?

Meet Esilaicos. The Social Network of your nightmares. Esilaicos has some brand new features no one wants.

  • Plain Text Messages.
    Want to talk to a single person? That’s great, because we have just the feature for you. PTM make it extremely easy to have a conversation with a person, including text and all. Because of administrative reasons, emojis cannot be used though, and the character limit is 140-20’000 characters. Nothing above, nothing below.
  • 4 PTM / Day!
    Our modern technology allows you to exchange up to four meaningful messages with a single contact every day. Still, this is a hard limit, so make sure every message counts! (Better read those messages twice, so you don’t need to crorrect yourself afterwards)
    *correct
  • Up to 100 Contacts!
    Our almost excessive limit makes it easy to talk to everyone you truly care about, and stay in touch with those new found contacts as well. Make sure things don’t get too cluttered, though!
  • Privacy!
    No more unwanted profile stalkers, no more unfriendly wall posts. Esilaicos is completely based around private conversations, making everything more personal and more private, and thus more enjoyable for both parties involved. If you feel adventurous, you can join our private Group Chat Beta and invite up to four other friends (due to technical limitations messages per person will be reduced to one per day)
  • Attention span!
    Your messages will be delivered every day at 13:00, regardless of when you press “Finalise”. However, messages can not be edited after pressing the “Finalise”-Button. A quicksave feature is of course available.
  • Profile Pictures!
    Since we are a state of the art technology company, we deliver a full profile picture implementation. Feel free to change your picture once every 180 days, and always keep fresh!

Want to sign up now?

I thought so. Of course, this is a bit of a hyperbole. Rather see this as a plea for a bit longer, more thoughtful messages, rather than just throwing a single letter into a conversation. I hope you had a bit of fun stumbling through all these letters, and congratulations on making it to the end (unless you cheated and just scrolled down, in which case I personally would recommend reading at least the part “tl;dr” for your own enjoyment and reflection).

Have an amazing day, and to all of you bloggers and authors out there: Keep writing those long texts. They are awesome.

k? k. :)

Installing a DNS-Server with Dnsmasq (also works on Raspberry Pi)

There are different reasons for why one would need a personal DNS server. May it be adding names to development projects (e.g. mysite.dev) when the page is hosted on a server that has no real domain attached to it, or if you just want to fool around with forwarding DNS queries, e.g. seeing how simple it is for your provider to log your data and all the accessed DNS requests. Regardless of what for you will use it – setting up a small DNS Server might come in handy.

Dnsmasq.

Dnsmasq provides the exact ability that we need for most projects: It “listens” to /etc/hosts while still remaining completely independent, and tunneling all the requests you might need. In this short tutorial, we’ll set up a machine of your choice to use the Google nameservers as resource, and then provide your own (forwarded DNS) via that very machine.
So let’s get started, shall we?

Setting it up

It’s rather easy to set everything up really, once you know what you have to do. Basically, you can just install dnsmasq via apt: apt-get install dnsmasq … Aaaaaand you’re done :D. I’m serious, you can now point any device of your choice in your home network to that IP address, and it’ll work on the spot. However, that makes no use. So let’s start out by modifying some files, to get everything up and working as we please.

It couldn’t be easier.

We can simply use everything that we use in our everyday life to manipulate queries that exit a machine on this one as well. Editing your /etc/resolv.conf by adding a line that says nameserver 8.8.8.8 will automatically switch dnsmasqs standard lookup address to 8.8.8.8 – whilst still looking for any altered routes in /etc/hosts. If you need fine tuning or want to look up how logging on a lower (verbose) level works, you should head over to the manpage, and find that special piece of information you are searching for. Whilst being super simple, with dnsmasq you get a powerful tool that enables you to modfiy queries in the way you need them to be.

This said, have fun modifying and logging traffic to all the outgoing DNS servers!

Oh, and a small hint for the paranoid people out there… instead of using a Google service and thus tunneling everything through that company’s servers, you can also try some of the DNS root servers – not only should they be updated earlier, but also are they better considering privacy. And don’t worry about availablity – trust me, these things are covered :)

Zurückgebloggt – Selfoss auf dem Uberspace.

Die Google-Reader-Ära ist vorbei
Google Reader ist tot – und so traurig das auch sein mag – manchmal muss man einfach weiterziehen, sich nach einer Alternative umschauen und dann damit weiterarbeiten. Glücklicherweise gibt es ja RSS schon etwas länger, und die Enthusiasten, die gerne ihren Feed-Aggregator auf dem eigenen Server laufen lassen möchten, können schon seit längerer Zeit auf Projekte wie Tiny Tiny RSS zurückgreifen. Hübsch war das aber nicht immer – und auch so ganz ohne Fehler hat’s bei mir wirklich nie geklappt. Sicherlich, so etwas ändert sich mit der Zeit, war aber trotzdem ein Anlass für mich, sich noch mal umzusehen, was denn noch alles in der Welt der Feed-Reader herumschwirrt.

Ein neues Sternchen am Himmel – Selfoss

Über drei Ecken und Kanten stieß ich dann auf das Selfoss-Projekt. Der schlanke Feed-Reader präsentiert sich in schlichtem Design und lädt sofort dazu ein, loszulegen. Besonders hübsch: Das Layout passt sich flüssig der Browser- bzw. Bildschirmgröße an und lädt so auch zum Lesen unterwegs ein. Schön ist hier auch, dass die WebApp-Funktionalitäten, insbesondere am iPhone, dazu verwendet werden, um einen möglichst schonen Vollbildmodus zu erhalten – da kann sich das Design durchaus mit manchen anderen nativen Apps messen!

Der Langzeittest wird zeigen, ob die Performanz unter der Haube auch so hübsch ist wie das Gewand, in dem der Reader daherkommt. Aber wie denn testen – wenn nicht erstmal installieren?

Installation auf dem Uberspace

Die Installation ist wie bei den meisten kleineren Projekten relativ einfach: Man lädt sich das ZIP-File aus der offiziellen Quelle herunter, packt es in den /var/www/virtual/user/url-Ordner und sieht unzip für eine halbe Sekunde dabei zu, wie der Ordner gefüllt wird. Jetzt gilt es nur noch das zu beachten, was sich beim Uberspace als Schwierigkeit herausstellen kann, wenn man denn nicht so affin im Umgang mit htaccess-Dateien ist. Aber keine Angst, auch das ist im selfoss-Fall kein Hexenwerk; es reicht, die Zeile RewriteBase: / in die verstecke .htaccess im Selfoss-Ordner einzufügen. Danach dürften übermäßig stürmische User, die sich vorher noch mit einem 500er Fehler bzw. Undefined index: SERVER_PORT auf der php-cli plagen mussten, auch die schön-schlichte Oberfläche von Selfoss sehen.

Damit die posts auch in regelmäßigen Abständen eintrudeln, dürfen wir nochmal kurz via crontab -e im Texteditor unserer Wahl einen Cronjob einrichten, der via curl oder wget http://[selfoss-url]/update holt. Nach dem ersten Indexing ist euer Newsstream dann auch schon gefüllt.

Und jetzt wird gelesen!

Alles eingerichtet, können unter den Sources neue Quellen hinzugefügt werden – aber nicht nur RSS-Feeds, sondern auch Künstler von DeviantArt oder die Heise News – alles nativ unterstützt! Ich für meinen Teil werde jetzt erstmal ein paar alte Subscriptions übernehmen – schließlich hat selbst dafür Selfoss ein geeignetes Interface.

Downloading the State Library

Weird things are happening out there. Our Bavarian State Library for example, refuses to let us download documents that are published under the public domain license. Me, being sneaky, wanting to get a workaround, have used up all Saturday on creating a cool workaround for what they consider safety in the internet. I’ll walk you through the process with the example of this document by Alfred Adler, published under Public Domain.

Protected by JavaScript™

When it comes to security, people at the Library seem to be very naive. But nevermind, it doesn’t get any easier. Open up Firebug or the Chrome Dev Kit, and remove the oncontextmenu="return false" in order to investigate further. Once that is done, we have a clear view of what is going on here. However, just investigating the HTML file we have lying around now doesn’t make sense as it contains links that seem not as clear as they could be. But what if we copy the direct address of the image? We get an URL like this: http://daten.digitale-sammlungen.de/~db/0006/bsb00067643/images/bsb00067643_00001.jpg. That doesn’t look that beautiful either. We could use it, but why not have a click on it? It magically transforms into something like this.

http://daten.digitale-sammlungen.de/~zend-bsb/wasserzeichen-projekte.php?seite=00001&id=00067643&webrumpf=/sub/db/public_html&bibl=bsb

Now, let’s analyze this further. I don’t think I need to mention that seite equals the page, and also, the ID seems quite similar to the unique identifier of the document, bsb00067643-7. But what if I want the whole document? Right, there’s scripting for that. I’ll choose PHP because I initially planned on porting this to a WebApp, and also because it’s easy to read. Let’s get started, shall we?

Getting started

In order to download a whole document, we need to download it page by page and then maybe zip it so you don’t have too many files lying around. But when to stop? We have to investigate this further in order to let the script end at some point. At this point, I have to thank my state library once again. It does not only just send a failed request, but also shows me their PHP errors. What I am going to do now is I’ll filter through if there is the word “Warning” in it, and thus I’ll end the loop. This however requires me to verify that the document exists before, otherwise I’d be stuck in an endless loop. But this is not about PHP scripting techniques, but the practical use case, and also about antique documents. Digging further into it, I figured that it might be a good idea to use Regular Expressions in order to filter out the ID. I must say that I hate Regular Expressions, and the only way I know around them is to try and error on websites such as rubular.

I have then come up with a solution that looks like this: bsb{1}(........) which hands me over the ‘bsb’ with the following numbers, disregarding the last ‘7’ which I still have not figured out what it is about.

Suddenly realizing that there is more to cut off in order to put that one into the link (I really wonder why I didn’t see that one in first place), I came up with the RegEx /\d{2,}/ which cuts everything away besides the numbers, two or more. Good thing is: If somebody puts in a link that contains a number starting with ‘bsb’, it still gets processed further. Unintended fail-safe, how much cooler could it get?

Self-explanatory practice

I don’t think I need to present you code right here as it is basically just two variables originating from the $_POST array (if you want to). I also made use of a Zip() routine that you can easily find on StackOverflow (hint, hint) – if you want to, as well.

Résume

I guess this short document shows the problems of when administrators try to protect documents that they do not even have the rights for. And then, which such failure in protection, probably originating from the 2000s, I don’t really know what to say. This just shows that if people want stuff, they get stuff. It’s that easy, probably. To a certain level, of course.

I still thank the people over at the State Library for making such an amusing ‘security system’ if could even call it one; and I had fun revising my coding practices which I kinda lost because of various other stuff there is to be done.

Two more things…

Please do not abuse library servers. They serve documents that might be useful for many people, and thus they really should be available at useful speed rates all the time.

Also, be sure to check your memory limit and your execution time settings in PHP. Otherwise, you might run in some nasty errors that blow up your script or stop the ZIP generation. But you’ll find out!

Cheers!