Example 6.2 is a write() method intended to be inserted into the metaweb.Session class of Example 4.15. It uses a number of the utility methods defined in that previous example, and the fact that these utilities can be shared between mqlread and mqlwrite code demonstrates how similar these two services are. In particular, recall that the _fetch utility method invokes the _http method, which does automatic cookie management. This means that our code does not have to explicitly handle authentication credentials or the mwLastWriteTime cookie. Also, remember that: _http automatically does an HTTP POST when a body is supplied for the request, _fetch does JSON parsing of the HTTP result, and _check raises a ServiceError if the code property of the response is not /api/status/ok.

def write(self, q, **options):
    """
    Submit the MQL write q and return the result as a Python object.
    Options specify envelope parameters. Authentication credentials, 
    from a previous call to login() must exist in the cookie jar.
    Raises ServiceError if the query fails.
    """

    # We're requesting this URL
    url = 'https://%s%s' % (self.host, WRITE)

    # These headers identify how the query is encoded in the request body
    # and guard against XSS attacks.
    headers = {
        'Content-Type': 'application/x-www-form-urlencoded',
        'X-Metaweb-Request': 'True'
        }

    # Gather options that apply to this query
    opts = self._getopts("use_permission_of", **options)

    # Build the query envelope, adding options to it
    envelope = {'query': q}
    envelope.update(opts)

    # JSON encode the envelope
    encoded = self._dumpjson(envelope)

    # Use the encoded envelope as the value of the query parameter in 
    # the body of the request.
    body = urllib.urlencode({'query':encoded})

    # Now do the POST and parse and check the response
    response = self._check(self._fetch(url, headers, body))

    # Return the result from the response envelope
    return response['result']

Example 6.1 and Example 6.2 are helpful methods, but they are just utilities, not real-world examples of how you might write to Metaweb. For the sake of example, let's suppose that you are a coin collector and you think that freebase.com should include information about each of the coins issued by the US Mint under its 50 State Quarters program. First, visit the US Mint's website to find out when each state quarter was released, how many were minted for each state, and when each state became a state.

Next, create a Metaweb type to model this data. Login to sandbox.freebase.com and create a new type named "US State Quarter" in your default domain. For our example, we'll use the type id /user/docs/default_domain/us_state_quarter.

Next, give your type four properties:

Optionally, make each of these properties unique by clicking the "Restrict to one value" checkbox.

Next, you need to get your data into manageable form. Extract data from the US Mint site, and arrange it in a plain text file named quarters.txt that looks like the following:

Delaware,1999-01-04,1787-12-07,774824000
Pennsylvania,1999-03-08,1787-12-12,707332000
New Jersey,1999-05-17,1787-12-18,662228000
Georgia,1999-07-19,1788-01-02,939932000
Connecticut,1999-10-12,1788-01-09,1346624000
Massachusetts,2000-01-03,1788-02-06,1163784000

Each line in this file is the data for a single quarter. Fields are separated by commas. The first field is the name of the state. The second and third fields are the release date and statehood date for that state. And the fourth field is the mintage for that state quarter.

With our type created, and the data in this format, we can now write a simple script to upload the data to sandbox.freebase.com. Example 6.3 shows how to use the metaweb.py module to do this. Note that you need to insert your own Freebase username and password into the script to make it work for you.

import metaweb         # Use the metaweb module

USERNAME = 'username'  # Put your Freebase username and password here
PASSWORD = 'secret'

# The ID for our US State Quarter type depends on our username
TYPEID = '/user/' + USERNAME + '/default_domain/us_state_quarter'

# Create a metaweb.Session object to interact with the sandbox server
sandbox = metaweb.Session("sandbox.freebase.com")

# Make sure we can log in before we go any further
sandbox.login(USERNAME, PASSWORD)

# We will be creating multiple quarters in a single MQL query.
# We start with an empty array and add MQL writes to it in the loop below
query = []

f = open("quarters.txt", "r")                  # Open our file of quarter data
for line in f:                                 # Loop through lines of the file
    # Break each line into fields
    fields = line.strip().split(',')           

    # This query creates a single quarter
    q = {'create':'unless_exists',             # Create a new object
         'id':None,                            # And return its id
         'type':["/common/topic", TYPEID],     # Make it a topic and a quarter
         'name': fields[0] + ' State Quarter', # The object's name
         'state': {'connect':'update',         # Connect to...
                   'type':'/location/us_state',  # a US State object...
                   'name':fields[0]},            # with this name.
         'release': fields[1],                 # Release date
         'statehood': fields[2],               # Statehood date
         'mintage': int(fields[3])}            # How many minted

    # Add this write to the array of writes
    query.append(q);

f.close()                                      # Close the data file

# Now send our one big query to Metaweb and get the result
result = sandbox.write(query)

# Display the id of the Metaweb object for each state
for r in result:
    print "%s %s: %s" % (r['create'], r['state']['name'], r['id'])

As we saw in Chapter 5, the MQL write grammar allows multiple independent writes to be specified in a single query as elements of a JSON array, and we took advantage of this fact in Example 6.3 to create all of our US State Quarter objects in a single invocation of mqlwrite.

Like mqlread, the mqlwrite service also allows multiple queries to be submitted using the queries parameter instead of the query parameter. To use the queries parameter, place one or more regular query envelope objects inside a new "outer envelope" object. The property names used in the outer envelope are arbitrary and are re-used in the response envelope. This is exactly the same for mqlwrite as it is for mqlread.

Suppose we want to create two objects in a single call to mqlwrite. Here are two envelopes that can accomplish that:

{
  "query":[{
    "create":"unless_exists",
    "name":"my test object #1"
  },{
    "create":"unless_exists",
    "name":"my test object #2"
  }]
}

{
  "q1":{
    "query":{
      "create":"unless_exists",
      "name":"my test object #3"
    }
  },
  "q2":{
    "query":{
      "create":"unless_exists",
      "name":"my test object #4"
    }
  }
}

When you include multiple writes in a single query, the writes are executed atomically: they all succeed or they all fail. As a result, they are not allowed to depend on each other, and there is no way to tell what order they are executed in.

If you submit multiple queries using the queries parameter, they are not atomic. Each one succeeds or fails on its own – the outer response envelope includes separate response envelopes for each query, and each of these inner response envelopes has its own code property to indicate the success or failure of the query. Note, however, that queries are unordered, and there is no guarantee that they will be executed in the order in which they are written. Since execution order cannot be predicted, queries passed to in the same invocation of mqlwrite should not depend on each other.

Note that the read() method of Example 4.15 accepts multiple read queries and uses the queries parameter to mqlread. The write() method of Example 6.2, allows only a single write query and uses the query parameter instead. It would not be difficult, however, to modify the write() method to use the queries parameter instead.

Each query envelope passed to mqlwrite includes a property named query to specify the MQL query to be executed. Any other properties within the envelope object are known as envelope parameters and provide additional input to mqlwrite. At the time of this writing ^[24], the only supported envelope parameter for mqlwrite is use_permission_of.

{
  "query": {
    "id":"/user/docs/music",
    "/type/namespace/keys": {
      "value":"notes",
      "namespace": {
        "create":"unless_connected",
        "type":"/type/namespace",
        "unique":false
      }
    }
  },
  "use_permission_of":"/user/docs/music"
}

import metaweb

username = "username"
password = "secret"
domain = "/user/" + username + "/music"

sandbox = metaweb.Session("sandbox.freebase.com", use_permission_of=domain)
sandbox.login(username, password)
sandbox.write({"id":domain,
               "/type/namespace/keys": {
                 "value":"notes",
                 "namespace": { "create":"unless_connected",
                                "type":"/type/namespace",
                                "unique":False }}})

The Metaweb mqlwrite service (and also the upload service documented later in this chapter) require a custom HTTP request header, named X-Metaweb-Request to be present in all requests. The value of the header is ignored, but if the header is not present in the request, the request will not be processed.

The requirement that the custom header be present is a security measure to prevent cross-site scripting (XSS) attacks. As a practical matter, it means that the mqlwrite and upload services cannot be invoked via HTML form submission, since there is no way to tell a web browser to add a custom header like this when POSTing a form.

For efficiency, Metaweb servers cache query results. Suppose you issue a read query and get a result. Then, another Metaweb user performs a write that alters the data you queried. If you re-issue your query, you are likely to get the same results (now cached) that you got the last time. After some time, the cached result will time out, and you'll see the new data written by the other user, but this will not happen right away.

If you're mixing reads and writes yourself, however, you always want your read queries to return the data you've just written, of course. Metaweb uses a cookie-based scheme to ensure that this happens. Any time you do a write with mqlread (or upload), the service returns a mwLastWriteTime cookie. Your web browser (or client code) then presents this cookie when it makes any subsequent read requests. This tells the mqlread service that it may not return any cached result older than the most recent write you've done. Your results may not arrive as quickly, but they will be consistent with the writes you've performed.

If you do your reads and writes using the freebase.com client, this cache management is invisible to you because your web browser handles cookies automatically. Similarly, the metaweb.py module developed in this chapter and Chapter 4 manages cookies automatically in the _http utility method, and the results returned by the read() method will be consistent with any previous writes made with the write() method.

If you develop your own library for working with mqlread and mqlwrite, you'll need to be aware of this caching issue and handle this mwLastWriteTime cookie appropriately.

Sometimes automatic cookie management is not enough to get proper caching behavior. Suppose you run one script to perform some writes, and then run another script to do some reads. Unless you saved the contents of cookie jar after doing the writes in the first script, and initialized the cookie jar of the second script from the saved state before doing the reads, you may get cached results that do not include the newly written data. Even trickier is the case where you do writes in the client (adding a property to a type, for example) and then do reads in a script. Unless you can make your script share the cookies of your web browser, you're likely to run into trouble. Another possible source of difficulty is when you perform reads in your web browser to verify the success of writes you've done in a script. If you check the state of an object with your web browser, and then run a script to modify that object, and then check the object again in your browser, you may not see the modification you've just made.

To solve these problems you need a way to obtain a current mwLastWriteTime cookie. The touch service does just that. If you make an HTTP GET request to /api/service/touch, the Metaweb server will send you fresh mwLastWriteTime cookie that allows any subsequent reads to see the current state of the database. At the time of this writing ^[25], you can do this in the freebase.com client by typing F8 to open the "Dev Tools" box at the bottom of the web page. Once you've done that, scroll to the bottom and click the "Refresh cache" link that has appeared. You can then type F8 again to remove the Dev Tools.

Example 6.4 shows how to invoke the touch service in Python. It defines a simple touch() method to be added to the metaweb.Session class of Example 4.15:

def touch(self):
    """
    Defeat caching by requesting a current mwLastWriteTime cookie.
    This method returns nothing.
    """
    self._fetch("https://" + self.host + TOUCH)

Despite its name, the value of the mwLastWriteTime cookie is not just a timestamp: it is an opaque value that contains cache information other than just the last write time. For this reason, it is not possible to simply create your own value for the cookie – you have to obtain a valid value from mqlwrite, upload, or touch.

Example 6.5 shows an upload() method for our metaweb.py module. It expects two arguments: a string of content (Python strings can be binary data or textual data) and the MIME type for the content. upload() creates a new /type/content object to hold your content and returns the id (in the /guid pseudo-namespace) of that object to you. This guid can be used to retrieve the content with the /api/trans/raw service (see Chapter 4).

def upload(self, content, type):
    """
    Upload the specified content (and give it the specified type).

    Return the guid of the /type/content object that represents it.
    The returned guid can be used to retrieve the content with 
    /api/trans/raw.
    """

    # This is the URL we POST content to
    url = 'https://%s%s'%(self.host, UPLOAD)

    # These are the HTTP headers
    headers = {
        'Content-Type': type,
        'X-Metaweb-Request': 'True'
        }

    # POST the request, parse the response, check for success
    # This will raise an exception on failure
    response = self._check(self._fetch(url, headers, content))

    # Return the id of the uploaded conent
    return response['result']['id']

In order to demonstrate the upload() method of Example 6.5 let's continue with our US State Quarter example. The US Mint has images of each of the state quarters on its website. Let's upload those images to sandbox.freebase.com, and make them visible through the /common/topic/image property of each quarter object. Example 6.6 shows how to do this.

In order to understand this code, you have to know that the upload service handles images specially. When you upload images, the /type/content object is also given the type /common/image with various properties filled out. Because your uploaded /type/content is co-typed as /common/image, you can link directly to image content from /common/topic/image.

import metaweb             # Our metaweb utilities
import urllib2             # For downloading images from the mint server

USERNAME = 'username'      # Put your Freebase username and password here
PASSWORD = 'secret'

# The ID for our US State Quarter type depends on our username
TYPEID = '/user/' + USERNAME + '/default_domain/us_state_quarter'

# Create a metaweb.Session object to interact with the sandbox server
sandbox = metaweb.Session("sandbox.freebase.com")

# Make sure we can log in before we go any further
sandbox.login(USERNAME, PASSWORD)

# All the images files are beneath this URL
imagedir = 'http://www.usmint.gov/images/mint_programs/50sq_program/states/'

# This dictionary maps state name to image name.
images = { 'Delaware': 'DE_winner.gif',
           'Pennsylvania': 'PA_winner.gif',
           'New Jersey': 'NJ_winner.gif',
           'Georgia': 'GA_winner.gif',
           'Connecticut': 'CT_winner.gif',
           'Massachusetts': 'MA_winner.gif'}

# Loop through the states
for state,filename in images.items():
    # First, download the image from the Mint's website
    image = urllib2.urlopen(imagedir + filename)
    type = image.info()['Content-Type']
    content = image.read()

    # Now upload it to Metaweb
    id = sandbox.upload(content, type)

    # Define a write query to link the quarter object to the uploaded image
    query = { 'type': TYPEID,
              'state':state,
              '/common/topic/image': { 'id':id, 'connect':'insert' }}

    # Submit the query and get the result
    result = sandbox.write(query)

    # Output the result
    print "%s: %s %s" %(state, result['/common/topic/image']['connect'], id)

Once you have run the code in Example 6.6, use the freebase.com client on the sandbox server to view your state quarter objects (refreshing the cache if necessary). You'll see that there are now images on the page.

It is fairly easy to upload an image and make it visible to users of freebase.com. It is a little trickier to do the same for textual content. When you upload a document, a /type/content object is created for that document. This allows the content to be retrieved with /api/trans/raw, but it doesn't allow it to be viewed in any natural way in the client. To accomplish that, you must create a /common/document object to reference the content, and (optionally) a /common/topic object to reference the document. Example 6.7 shows how you can do this.

import sys, re, metaweb 

# Read the content of the file specified on the command line
# It must be an HTML file with a <title>
filename = sys.argv[1]
try:
    f = open(filename)
    doc = f.read()
    f.close()
except Exception, e:
    sys.exit(e)

# Search through the document for a title
try: title = re.search("(?i)<title>(.*)</title>", doc).group(1)
except: sys.exit("Document has no title")
    
# Log in to the sandbox server. 
USERNAME = 'username'    # Put your own username and password here
PASSWORD = 'secret'
sandbox = metaweb.Session("sandbox.freebase.com");
sandbox.login(USERNAME, PASSWORD)

# Upload the document content to Metaweb.
# Note that we hardcode the text/html content type.
content_id = sandbox.upload(doc, "text/html")

# Submit a MQL write query to create a /common/topic and /common/document
# for the uploaded content
result = sandbox.write({'create':'unless_exists',
                        'type':'/common/topic',
                        'id':None,
                        'name':title,
                        'article' : { 'create':'unless_exists',
                                      'type':'/common/document',
                                      'id':None,
                                      'content':content_id }})

# Tell the user what we did
print "Uploaded %s: %s\n\tcontent: %s\n\tdocument: %s %s\n\ttopic: %s %s" % (
    filename, title, content_id,
    result['article']['create'], result['article']['id'], 
    result['create'], result['id'])

This Python program expects the name of an HTML file as a command-line argument. It reads the file and determines the document title by searching (using a regular expression) for a <title> tag. It uploads the document text with the upload() method of Example 6.5. Then it submits a MQL write query to create a /common/topic that refers to a /common/document that refers to the uploaded content. It uses the document title as the name of /common/topic object.