HOW Social Fixer WORKS
(And Why It Sometimes Doesn't)

Share on Facebook

Over 300,000 active users a day benefit from the features of Social Fixer. But have you ever wondered how it works? And why it sometimes doesn't? This should explain a lot...

The Basics

Social Fixer (SFX) is a "user script" written in Javascript. It was originally written for Greasemonkey, and has since been tweaked to run in other browser environments.

Once a Facebook page loads, the SFX script is loaded and executed. It then has access to everything visible on the page, and can manipulate it however necessary. It scans the page for things to change, and adds its own content into the page as well. SFX only runs once - on page load.

SFX does not have access to the server-side resources. This means it cannot run queries, or load data that is not available on the page, or store its settings in the user's account. For the most part, if the data isn't on the page, then SFX cannot use or manipulate it.

Handling Dynamic Content

Facebook has a very complicated navigation scheme. When you click links and navigate around the site, it doesn't load a whole new page like most sites. Instead, it runs Facebook's javascript code that loads new data and replaces some of the content already on the page. But when this happens, SFX doesn't get to run again to process the new content, because the page hasn't been reloaded. Instead, SFX needs to "watch" for new content and respond to it.

It does this by inserting an event listener on the window's "DOMNodeInserted" event. Every time Facebook inserts content into the document, SFX inspects it to see if it needs to react and do something. So when posts are added to the stream, SFX can inspect the content, see if it is a post, and then process it. Unfortunately, Facebook inserts a lot of content. And SFX needs to inspect everything, because there's no way to predict what will be inserted when. This is the primary reason for the slight slowdown when using SFX - it is processing a lot of content in the background.

Adding Content

In order to add new content to the page, like panels for "My Pages" or "Friend Activity", SFX needs to wait for the container to appear on the page. When Facebook paints its page, it is actually quite empty. Most of the content and structure is added dynamically through javascript after the page loads. So when SFX first runs, there is no content area, or right column to insert content into. SFX needs to wait for the right column to appear (using the DOMNodeInserted event as described above) and then insert its content.

This is accomplished by identifying the unique HTML ID that Facebook assigns to the right column. When that ID is loaded, SFX can go ahead and insert its content. Unfortunately, Facebook changes the structure of its page and its internal ID's randomly. So if SFX is waiting for a DIV with the ID "rightCol" to be inserted, and Facebook now calls it "right_column", then SFX will never find the container and will never insert its content. So SFX is very dependent on the structure of the HTML returned by Facebook. Much time is spent reverse-engineering their code to find the right places to hook into.

Getting Data

SFX displays a lot of stuff on the page that isn't already there, though - lists of My Pages, My Groups, Friends Activity, Friend Tracker, etc. Where does it get that information from?

This is even more complicated. There are calls that the Facebook page itself makes back to the server to get data - for example, when you start typing in the search box. If you type an "a", then Facebook calls back to its servers to get a list of your friends, pages, and groups that begin with "a". Through a lot of trial-and-error reverse engineering, I've examined these requests and I can simulate them myself in SFX code. I then have to do some custom processing on the result, but what SFX ends up with is a list of your friends, pages, groups, etc. This information is stored by SFX and is used to display in the lists on the left panel, it's used to make the select lists in the Filters tab of Options, and it is used to detect unfriends.

When raw data isn't available like it is in the case above, then SFX needs to resort to another trick - it creates a hidden IFRAME that points to a Facebook url that has some data it would like to have. When the page loads in the hidden iframe, SFX parses the content and grabs what it needs, then destroys the iframe. This is how Notification Previews work on hover - in the background, SFX is loading the page that displays the post being previewed, cuts out just the post and comments, and displays them. This is also why, unfortunately, there is some delay in showing the previews.

Storing Preferences

All data that SFX remembers is stored in the browser itself. Nothing is stored on the SFX server, and certainly nothing is stored on the Facebook server. This is why SFX settings and data do not "follow" you from computer to computer, or even from one browser to the next on the same computer. SFX is not attached to your Facebook account. It's attached to your browser. All its settings and options are stored in whatever facility your browser provides to store data.

Since SFX started as a Greasemonkey script, it by default uses the GM_setValue() and GM_getValue() calls to store data. In Firefox, this reads and writes data to a file that the browser reads when it starts up. These settings can be found in about:config. In Chrome, Safari, and Opera, this storage space doesn't exist, so instead there is a wrapper around these calls that uses the localStorage area provided by these browsers.

Processing/Filtering Posts

Thankfully, Facebook delivers posts with special "meta-data" attached to each one. This is the root of how SFX can allow filtering, tabbing, etc.

In the HTML source, there is an attribute on each post that looks like this:

data-ft:{"src":10, "sty":46, "actrs":"14385334364", "pub_time":1289830690, "fbid":"1485431831867", "s_obj":11, "s_edge":1, "s_prnt":11, "pos":1, "sec":"new", "filter":"lf", "app_id":"201278444497"}
This is the data we need!

SFX parses this when processing each post and extracts the data.

  • sty = Story type. Each type of story, like wall posts, status updates, pictures, links, etc has a unique story type with its own number. Unfortunately, these are not documented anywhere! I have to figure out the types by observation and trial and error. It's painful. But knowing this type numbers allows SFX to do filtering based on what kind of story it is.
  • actrs = The unique Facebook id's of the person (or people) that made the post. Again, good for filtering.
  • pub_time = The time the post was made. This is useful later...
  • fbid = The unique Facebook ID of the post. Every post has its own ID. At least, it should. See the explanations below for why this is not as reliable as it sounds
  • app_id = The unique ID of the Facebook application that made this post
The other stuff is either unimportant or I don't know what it means!

When each post is inserted into the document, SFX examines this data and compares it to its internal data storage to see what to do with it. If the FBID has been seen before and marked read, SFX adds a class to the post that hides it. It looks at the app_id (if it exists) and compares it to its internal list of known applications, as well as the filters, and decides if it needs to move the post to a new tab or process it somehow. This is also the time when the action icons are added to each post.

When you mark a post as read, SFX grabs the FBID of the post and updates its data about the post - that it has been read, when this was done, and how many comments the post had when marked as read. This data is then used the next time the page loads, to decide whether the post should be hidden, and also to compare the new number of comments with the old count, to see if there are any new comments which should make the post be visible again.

UPDATE: Facebook recently stopped providing a lot of the data that used to be found in the data-ft attribute. So Social Fixer can no longer identify the type of post, the authors, etc. I've been able to find other ways to get some of this information, but not others. This is why I can no longer offer filtering by post type, because they don't expose that information anymore. This is really frustrating - entire chunks of information that I based a lot of functionality on can suddenly disappear without warning. What Facebook giveth, Facebook can taketh away ;)

CSS

CSS is short for Cascading Style Sheets. These are basically the rules that tell the browser how to display the page - colors, fonts, borders, whitespace, backgrounds, etc. Modern web sites make extensive use of CSS to style their page, and Facebook is no exception. Facebook uses thousands of style rules, hooked into hundreds of unique ID's and classes attached to the HTML elements.

SFX uses these ID's and classes to attach its own CSS rules that can customize how the page is displayed. By inserting new rules, SFX can hide things on the page, add borders, highlight things, etc. When you choose to hide panels on the ride side, like Pokes for example, SFX inserts CSS that hides the panel with a specific ID.

Much time is spent inspecting the Facebook source code to identify the structure, ID's, and classes used in order to create the correct CSS rules to customize the page.

Supporting Multiple Browsers

Almost all of the code written for SFX is the same for all browsers. However, there are some quirks and differences. It takes a lot of work to figure out all the unique behaviors of browsers and support them equally with the same code.

In fact, there are some things that require separate code for different browsers. In this case, SFX either branches internally or another approach is needed - the code itself has to be different depending on whether it's running in a Greasemonkey script, a Chrome extension, or an Opera extension. To accomplish this, I have built a complex, automated packaging system. The real source code contains special comment structures like /*CHROME (code here) /CHROME*/ that indicate to the packaging script that the code in that block is only to be used in the Chrome extension.

When SFX is packaged for release, I run the automated packaging script and it takes my real source file and chops it up using Perl and creates 8 unique versions of SFX:

  • Firefox Greasemonkey
  • Firefox Add-On for distribution on socialfixer.com
  • Firefox Add-On for the Mozilla directory
  • Chrome Extension for distribution on socialfixer.com
  • Chrome Extension for the Google Chrome Directory
  • Safari Extension for immediate distribution
  • Safari Extension that is pending approval by Apple
  • Opera extension
The code in each of these versions is slightly different. For example, Chrome does not permit SFX to make AJAX calls back to socialfixer.com to check for new versions. So instead, it needs to dynamically load a <script> tag to insert code from my site into Chrome. This kind of code is not permitted by Mozilla in add-ons hosted on their site, so it can't even appear in the source, otherwise they will reject it. So the packaging script automates all these rules and different code to make distribution less painful. It used to take me a half hour to package up a release. Now it takes me 5 minutes.

Why Does It Sometimes Not Work?

Of course, nothing is perfect. SFX can randomly break sometimes, or even be very frustrating by simple not working for some people. This is very frustrating for me as the author who spends many hours trying to keep it working correctly. Here are some of the reasons why this happens..

Randomly Changing FBIDs
As noted above, the FBID is what uniquely identifies posts, and allows SFX to know if the post has been seen before, marked read, etc. Unfortunately, Facebook sometimes just randomly changes the FBID for no good reason (that I can figure out). So you may have marked a post as read, but it will appear again in your feed because it has a new FBID.

This happens mostly with certain types of stories, so there is code in SFX to try to take it into account. Instead of uniquely identifying a post by the FBID, SFX will try to combine the story type, actor, pub_time, and/or object_id to create a unique string. This works most of the time, but not always. Unfortunately, if Facebook doesn't deliver a reliable way to uniquely identify posts, it makes it very hard for SFX to work.

Missing data-ft
Sometimes, the entire data-ft structure that holds post meta-data is just missing. This seems to happen randomly, and lasts up to an hour for most people. It often happens on Sundays, and may coincide with Facebook rolling out code changes. But the result is that SFX cannot get ANY information about posts in order to process them. This results in all posts showing, nothing being sent to tabs, and "mark read" having no effect. It seems like SFX has completely broken. But things return to normal once the data-ft information returns (which it always does).

Browser Storage Problems
Storing all options and post states in the browser's preferences isn't ideal, but it's necessary. Unfortunately, this often causes problems because browsers can be unreliable with storing data. Sometimes, it tells SFX that data has been stored, when it actually hasn't.

In the case of Safari, localStorage is used to store data. Unfortunately, there is an internal limit to how much data can be stored there, and it is stored per web site. And I've seen cases where Facebook itself writes tons of data to this storage place, effectively filling it up and preventing SFX from writing anything! In this case, I try to detect this problem and clear out Facebook's clogging data, but it doesn't always work. This can result in the dreaded QUOTA_EXCEEDED_ERR: DOM Exception 22 error. I cannot control this. It's the browser being stupid.

As if all that wasn't frustrating enough, Safari also keeps its localStorage per HOST, rather than per DOMAIN. So when you are on www.facebook.com you get one storage area, and when you are on apps.facebook.com you get a different storage area. So it looks like your prefs are not sticking when you save them! This is very frustrating, and caused me to simply turn off SFX functionality on any host but www.facebook.com.

Conflicts With Other Apps
Sometimes other applications are installed that cause problems with SFX. This could be because they are trying to process the page also (true of other Facebook add-ons or scripts) and when two scripts both try to manipulate the page it can cause endless loops or weird behavior. It is also caused by some apps like AVG LinkScanner, which apparently tries to protect you from bad links but because of the way SFX inserts content and manipulates the content on Facebook, causes AVG LinkScanner to slow the user's computer to a crawl.

These conflicts can often result in your browser becoming really slow or unresponsive. I try to address these conflicts where I can, but unfortunately there is just no way to know about every app that might conflict and try to avoid it. For the most part, this is usually a problem with the other apps, not SFX.

Facebook Changes
Facebook is known for randomly changing their functionality, layout, and design. Features will be there one day and gone the next, and users often think that SFX is the cause.

When they do make changes, the structure of the page often changes as well. So the ID's and classes that SFX depends on to work correctly can change, and SFX can no longer find the containers it needs to add content, or it can't parse the pages it gets data from. In these cases, a new release of SFX is needed after I can inspect the changes, reverse-engineer the new stuff, and make code changes to consider it. Though this annoys some people (too many updates, they say!) it is necessary because of the way Facebook changes its code.

Remember, SFX is not using official, published API's that are documented and consistent. It uses undocumented, evolving, randomly changing code that is delivered at the whim of the Facebook developers.

Facebook's Navigation Scheme
Usually on web sites, when you click a link it navigates to a new page. With Facebook, though, it usually doesn't. Instead, it changes the url in your browser and adds something at the end after a "#" mark, and internally the Facebook code requests new data and updates the page with it. So your browser hasn't completely re-loaded the page. Facebook has just modified the contents with new stuff - even if it's a whole new page!

This causes a lot of problems. Since SFX only gets to run once - when the page loads - then it doesn't get to run and see that you are on a new page. Also, it can't inspect the url to see what page you are on, because the url doesn't necessarily accurately represent the page being displayed. So instead, SFX has to try to detect all these internal page changes and respond appropriately. Rather than just adding panels to the right side when the page loads, it needs to watch for any time the right column gets created, and try to insert its content then. This results in s real mess of coding inside SFX, because nothing can be known for sure.

It also causes things to sometimes break. If SFX doesn't properly detect that things on the page have changed, it may not re-insert its content correctly. The symptoms of this are often people saying their control panel has disappeared, or they get duplicate copies of "Friend Tracker" or something. This is because Facebook's internal navigation has confused the SFX code. I have spent many, many hours trying to perfect a system that detects and accounts for this crazy navigation scheme on Facebook. It's still not perfect, and is one of the biggest sources of pain for development.

Crazy Code Caching
Since SFX runs in the browser, right along side Facebook's code, the SFX code is at the mercy of what Facebook decides to do. And in one case, it does some crazy stuff.

After the page paints, Facebook's code will (for only some browsers, for some reason), inspect the content of the page and store it in its own data. Then if you navigate away from the home page to a profile for example, and then click the Facebook logo to return to your home, it will re-insert this stored content to the page, rather than getting fresh content.

Seems simple enough, but unfortunately this content also sometimes includes the control panel that SFX has inserted. And once Facebook re-inserts it, all the functionality connected to the buttons is stripped away! So now the control panel exists again, but nothing works. And SFX can't detect that this happened, because it doesn't get to run again. And if it looks at the page, it thinks the control panel is already there, and working correctly. Confusing!

Duplicate Posts
For some reason, when Facebook delivers content it will often deliver duplicate posts. The same story appears more than once in the feed! This confuses users, and they think it's a SFX problem, so SFX has code to detect duplicates and hide them. However, detecting duplicates depends on the ability to uniquely identify posts, which as mentioned above does not always work correctly. Doh!

Conclusion


Share on Facebook

I hope this page has given you some insight into how Social Fixer works, and some of the frustrating factors that make it not work sometimes. If you have any comments or suggestions or questions, please post to the Social Fixer Page on Facebook.

I hope you continue enjoying Social Fixer!