Contributing Large Datasets

Contributing Large Datasets: NNDB Links for People

2008-09-05T21:34:58.0012Z

Thanks Brian. The exact count should be 19,619 writes to the API with 2 properties being updated each time.

Contributing Large Datasets: NNDB Links for People

2008-09-05T17:33:09.0000Z

I believe the normal limit is 10,000 writes. I'll see if I can get your limit increased.

Contributing Large Datasets: NNDB Links for People

2008-09-05T03:28:16.0012Z

I've uploaded a new version of the data to the sandbox. If no one has any objections, I'll add it to the main site.

Is there a limit on how many writes I can do on the main site? Will I be able to make 19,000 writes in a day?

Contributing Large Datasets: NNDB Links for People

2008-08-21T03:06:20.0012Z

Looks like everything is working fine now in the new release. Thanks for fixing this.

Contributing Large Datasets: NNDB Links for People

2008-08-15T18:01:15.0012Z

Ok, thanks guys. I'll watch for CLI-4538 in the release notes.

Contributing Large Datasets: NNDB Links for People

2008-08-15T16:35:59.0012Z

This sounds like simply a bug - you're right that the NYTimes links got added separately, and they'll probably need to be fixed. It may be too late to get a fix in for next week's release, but I'll try. For my and other's reference, this is CLI-4538 in our bug system.

Contributing Large Datasets: NNDB Links for People

2008-08-15T16:34:26.0012Z

Ah, yes, the char escaping with keys and URI templates... I am in the process of converting the NYT keys to use URI templating, and I also ran into that problem. There's a bug filed to have the UI behave properly when it encounters escaped URLs - I'll post back when there's a status update to this.

Toby (tsegaran) actually added a key, and created a discrete weblink (it's not using URI templating). When I implement the URI templating, I'll be removing the superfluous weblink.

BTW, good work!

Contributing Large Datasets: NNDB Links for People

2008-08-15T15:29:03.0012Z

Ok, I've created the namespace, I've set up the enumeration property on the NNDB Person type and I've attached a URI template to that property.

Then I added a sample key to the Paul Newman topic and the NNDB link shows up as expected. Unfortunately, the ID has a forward slash in it which gets escaped and breaks the link. I went back to the Foreign Key Property and explicitly disabled URI encoding but that hasn't fixed it. Any ideas on how to handle this?

Coincidentally, tsegaran added a foreign key to a NYT page on the same Paul Newman topic and his key also contains forward slashes but he seems to have entered the weblink seperately without using a URI template.

Contributing Large Datasets: NNDB Links for People

2008-08-11T16:34:01.0000Z

Oh, yeah! It’s totally done and I forgot to come back here.

First, I made namespaces (/authority/imdb/film, name, character). Then I made properties that enumerate those namespaces and attached URI templates to them.

Check out the IMDb profile property on film. It expects Enumeration as its type. Then you have to get a little fancy and switch to the admin view. Set the Enumeration property of the property to the expected namespace. Then co-type the property as a Foreign key property and set a URI template. Set its type to URI Template and fill in both the canonical template (used to generate and recognize URIs) and the other templates (used only for recognition).

The schema UI will support this at some point… just not yet, as it’s kind of a power-user thing.

Contributing Large Datasets: NNDB Links for People

2008-08-11T05:35:53.0012Z

Have you made any progress on this? I've looked at the documentation on enumerations but I can't figure out how to apply it to my model.

Contributing Large Datasets: Link airports as containedby for apropriate locations

2008-06-03T16:16:20.0007Z

FYI - We are trying to get our hands on some country/airport data also, so if licensing is compatible, we may be able to add country containment to a large percentage of airport topics, and perhaps and new ones!

Contributing Large Datasets: Link airports as containedby for apropriate locations

2008-06-02T23:49:51.0007Z

You have a point, I think there is no airport that is actually in the city they serve, it would be just outside the city. But nobody says thei're flying to schiphol rijk or whatever (unknown small) town that is closest to the airport or containing the airport.

I guess thats why serves is there. In case of Schiphol, the administrative division Amsterdam is responsible for Schiphol.

I will only add the countries containing the airport.

Contributing Large Datasets: Link airports as containedby for apropriate locations

2008-06-02T17:39:24.0007Z

Is Schiphol actually contained by Amsterdam? I am not that familiar with Dutch location containment, but it seems like Schiphol is contained by the municipality of Haarlemmermeer and possibly contained by the city Schiphol-Rijk.

Are you just adding containment by country, or containment by administrative division also? If it is also the latter, there could also be the situation where an airport may serve one location, but be contained by a different administrative division. The example I'm thinking of is Newark Airport is contained by New Jersey, but serves New York City.

Contributing Large Datasets: Link airports as containedby for apropriate locations

2008-06-01T13:08:42.0005Z

Airport is co typed as location so I would like to set airport as the containedby value for the correct locations.

For example Amsterdam Schiphol should be contained by Amsterdam and Netherlands.

Got some script and run it on the sandbox, it will get the airports serves location and add the airport as contains value for the serves location and all locations that contain the serves.

Like Amsterdam Schiphol serves Amsterdam so schiphol is added as a value contains for amsterdam, amsterdam is contained by netherlands so schiphol is added as contains value for netherlands.

Contributing Large Datasets: Postsecondary Schools

2008-05-07T23:35:03.0008Z

The types you should be looking at are /education/institution and /education/university. Between them, they should have all the properties for post-secondary schools. (Educational Institution has properties that are common to schools of all levels.)

If there is data in your dataset that you want to load,and which doesn't match any existing properties, we can talk about adding new properties. The first place to start, though, would be to create a type in your private domain to hold those properties so you can test them out and so that people can review them before adding them to the types in the education domain.

This looks like a great dataset. Feel free to keep posting questions while you work on it.

Contributing Large Datasets: Postsecondary Schools

2008-05-06T01:08:04.0005Z

I'd like to get the list of all postsecondary schools loaded.

The nasty format is here: http://www.ed.gov/offices/OSFAP/PEPS/dataextracts.html and I can get it into a better format but the current definition of "school" in freebase leaves a lot to be desired.

What should I do?

Contributing Large Datasets: Food Recipes

2008-04-30T20:50:21.0007Z

Hi Sebastian,

User skud was looking for reviewers on her food domain. I would think that would be a good start for then developing a type for recipes where it could use those types. Perhaps you may want to collaborate with other users on creating a recipe type? You could start a discussion on the data-modeling list (if you're not part of the list, you can join here).

Contributing Large Datasets: Food Recipes

2008-04-30T16:51:37.0000Z

Link is broken or leads to nothing?! is there any solution adding recipes in freebase?

sebastian

Contributing Large Datasets: 'Similar artists' data for musical artists

2008-04-04T07:33:48.0000Z

Ugh, right, I forgot to check the NonCommercial part. Never mind...

Contributing Large Datasets: 'Similar artists' data for musical artists

2008-04-03T23:02:03.0007Z

Thanks, Jeroen. First, please be sure that the data can be contributed legally; we did not collect the artist similarity information from MusicBrainz, for instance, because it is licensed CreativeCommons-Non-Commercial, and as a commercial enterprise, we can’t legally use that data.

If you have permission to load that data, then this sounds like a great idea! You could also add the last.fm page as a Web link. For lookup, since last.fm and Freebase both use MusicBrainz keys, a key is probably not needed

Contributing Large Datasets: 'Similar artists' data for musical artists

2008-04-03T11:02:33.0005Z

Hi there,

I have a data set with for about 38,000 artists with a MusicBrainz ID a list of the artists that are most related to it according to last.fm.
I have about 60,000 more without a MusicBrainz ID and I can obviously retrieve more data through the last.fm webservices.

My plan is to add all artists with a MusicBrainz ID that are more than 80% similar to an artist (about two or three artists, usually) as a 'similar artist' relation. Is that ok?

As an example, I added links for about 100 artists to the sandbox. I do my lookups based on the MusicBrainz ID, and do not create new artists. See: Édith Piaf

Any comments? Should I somehow link the artists to their last.fm page after processing? Add their last.fm urlname as a key, perhaps?

Thanks,

- Jeroen

Contributing Large Datasets: Currency exchange rate

2008-02-22T09:01:42.0000Z

I've subscribed to developers now. Thanks for the detyping tip. I de-typed all the entries and imported 3 datasets (1971-1989, 1990-1999, 2000-2008) for USD-AUD into sandbox and everything looks ok. I did get a 400 error about midway into doing 1971-1989 but I just restarted it from the last entry that was added and it worked out.

Contributing Large Datasets: Currency exchange rate

2008-02-20T01:00:31.0007Z

There's no quick way to delete anything. You could de-type them, though, which would get them out of the way of your scripts, at least. Questions about scripts, MQL, and the API can be posted to the Freebase developers' list, also, where you'll probably get a quicker response than the 17 hours it took for this (sorry! my RSS reader lost this message for some reason): http://lists.freebase.com/mailman/listinfo/developers.

Contributing Large Datasets: Currency exchange rate

2008-02-19T07:59:52.0000Z

I attempted an import on sandbox today.. Seems to have not worked out so well. After doing a lot of reformating on the date to get it into a form that the API would accept, I ended up getting a 503 every time I tried to do the import. I didn't seem to see anything on sandbox's web interface so I kept trying. Much later I looked and there were records, but It didn't look like they'd all been imported, so I made my import script do the imports in 100 unit segments, and started to get a 2 entries match error, so I looked that up and discovered that I had duplicate records even though I'd use unless_exists in my create clause! I'm not exactly sure what to do at this point, is there a way to delete all the records in a type on sandbox?

Contributing Large Datasets: Currency exchange rate

2008-02-18T23:18:31.0007Z

I think that because the source and target are explicitly modeled there is no need to add the reciprocal rates since an application can just reverse them.

Contributing Large Datasets: Currency exchange rate

2008-02-18T20:59:29.0000Z

I've moved "exchange rate" to the finance domain. This change will be copied to sandbox tonight, so you will be able to try our your import there. I'm not sure about the reciprocal exchange rates, but I'll ask around.

Contributing Large Datasets: Currency exchange rate

2008-02-15T05:38:57.0007Z

I looked at the currency pages and they look fine, and the schema looks fine, so I'm going to assume it was connected correctly. I've been trying to figure out if we want to do reciprocal exchange rates, since with AUD/USD I can calculate USD/AUD easily, would it be a good idea to add both records, or to just let people do the calculations themselves?

Contributing Large Datasets: Currency exchange rate

2008-02-15T00:46:43.0007Z

I've connected "exchange rate" to the currency type; it was more complicated to do than I expected, so please take a look at the types and let me know whether I did it correctly. If it's set up correctly, we can move the "exchange rate" type to the finance domain.

Contributing Large Datasets: Currency exchange rate

2008-02-14T21:29:45.0007Z

That was what I originally wanted to do, but realized I obviously couldn't. Having made exchanged currency though, I'm not 100% sure that it's not a better option now. The basic idea is that not all currencies are exchanged. On the other hand, I think just having a blank target/source field may be enough to let you know it's not exchanged :)

Contributing Large Datasets: Currency exchange rate

2008-02-14T18:36:23.0007Z

You've generated a lot of interest here over your dataset, and we'd like to work with you to put your types in the top-level /finance domain. What we probably would do is hook the "exchange rate" type to the existing "currency" type, rather than keeping the "exchanged currency" type. What do you think?

Contributing Large Datasets: Currency exchange rate

2008-02-14T01:32:36.0000Z

Right now, I've got USD-AUD from 1990 to 1999, which is 2561 records. I'm getting the data from: http://www.federalreserve.gov/releases/h10/Hist/default1999.htm I have a script ready to go to do the import of the AUD-USD dataset, and afterwards I'd like to import more recent data, and then start to import other currencies from the reserve banks datasets.

Contributing Large Datasets: census data

2007-09-01T09:14:34.0000Z

I did not get a notification of this response, even though I am watching this thread. Thus my delay in response.

Good to know you can assist. Look forward. What next?
Let me know how we can communicate further. Shall I upload a few files of census data somewhere and make it available to you?

BTW, I have not found an discussion on US census data on freebase. Can you give me a link? In fact I wanted to look at any census data type but could not locate them (on freebase).

d

ps: cc replies to my email if you can

Contributing Large Datasets: Food Recipes

2007-08-23T18:12:13.0007Z

Hi Kiril-

There was some discussion in the "Food" domain awhile back about adding recipes. You might want to post something there and see if anyone else is starting to work on it. You can read that discussion here.

Contributing Large Datasets: Food Recipes

2007-08-23T01:46:44.0012Z

Hello, I have found that there are a lot of recipes available in the public domain as mealmaster format or many times as RML (recipe xml files). I have wrote a script to convert those formats and insert them in my sql database. The thing is that I am sure this is a very valuable information for a freebase and it will also help me write my application without needed to store myself all the recipes.

The list of recipes I am talking about is here: http://dsquirrel.tripod.com/recipeml/indexrecipes2.html
Please, take a look at on of the files and tell me if this is valuable information for Freebase and I will start experimenting in sandbox.freebase.com and see how the data will be stored. Also I did not see a category for recipes here on freebase and I am thinking if you wanted to include it at a later stage or what?

Thanks,
Kiril

Contributing Large Datasets: census data

2007-08-22T17:54:14.0000Z

dinesh - This sounds like a very interesting data set. As you may have noticed we have been somewhat selective in our use of the US Census data, trying to identify the level of detail that is useful to people as well as integrating the Census data across Freebase domains. I'd be very interested to work with you to select and map the India Census fields as well as cull the appropriate level of detail from the records (for instance, we have deliberately omitted the "block" level of detail in the US Census.)

Contributing Large Datasets: census data

2007-08-22T12:18:38.0005Z

We would like to use freebase to store census data of India. There are over 120 fields and many thousands of records. Say over 300,000 records. Our idea is to also add GIS info for these records eventually. For now, it would be great if we can upload a few thousand records, which are available as xls files.

Contributing Large Datasets: Nonprofit Organizations

2007-08-03T22:27:23.0000Z

If you're a newly registered user, then unfortunately, yes, you've guessed correctly. User registration data is copied over from freebase.com to sandbox.freebase.com once a week (usually on Monday afternoon). Sorry - we hope to improve this in the future.

Contributing Large Datasets: Nonprofit Organizations

2007-08-03T22:09:22.0011Z

I'd like to start work on the IRS tax-exempt organizations, but Sandbox is not allowing me to log in. Do I have to wait for Monday when Sandbox is refreshed.

Contributing Large Datasets: Economic Time Series Data

2007-07-19T12:04:15.0000Z

Hi Bob, I'm interested in time series as well, and have initiated a discussion of freebase support for them associated with the Data Modelling entry in the help pages. fyi, I'm developing a wiki environment for collaborative analysis of public domain data, especially time series. I'm investigating the possibility of using Freebase as the data engine. Regards, Mike

Contributing Large Datasets: Economic Time Series Data

2007-07-18T19:52:30.0007Z

Hi Bob- The best way to get something like this going is to create the data types in your private domain, then try loading a small subset of the data to see if your model is working or not (you don't need to dump it all up there). Getting the model right is actually the hard part. :-) Once you're happy with the model, add a discussion post to the domain you think your data would be most appropriate for, and the admins can help you take it from there. And feel free to post any questions that come up during the modeling process. Good luck - and have fun!

Contributing Large Datasets: Economic Time Series Data

2007-07-18T15:48:53.0004Z

I've just signed on with an interest in quantitative economic data (and other stuff, but let's start there. ) First, I don't see any such data available; a query only produced a definition. Since much of this is government prepared it is public domain and I'd like to begin uploading it with currently active time series and some interesting historical series. My vision was that if we had a fait number of others involved we could easily allocate the tasks of regular upload of current data as well. I'm not a developer, so I need guidance, but I do know data and where it is available in a public domain format. Let me know how to proceed (and when if it is still too soon). Of course, if I've missed it and this kind of data is already there, let me know. Thanks, Bob A

Contributing Large Datasets: What's the best way to do that?

2007-03-13T00:38:43.0000Z

either is fine. Thanks!

Contributing Large Datasets: What's the best way to do that?

2007-03-12T19:50:03.0006Z

Is the best way to do that just by sending an email to alpha@metaweb.com?