<http://trac.webkit.org/changeset/65351> has broken the ability to build WebCore in production configurations. The change to DerivedSources.make attempts to access directories outside of the WebCore source directory. Projects are built independently and without access to any other source trees. This needs to be fixed as soon as is possible as this change has completely broken the ability to build production builds of WebKit on Mac OS X and Windows. If it cannot be fixed promptly we’ll need to roll this particular change out.
Seems easy enough to fix. Rolling out the change will just break your builds in other ways.
A second issue is that the script involved uses third-party Python modules. That will also need to be addressed.
The 3rd party python module "simplejson" is already provided in WebKitTools, which is why the tool was placed there. When I placed the tool there, I was not aware of Apple's build limitation. Thank you for bringing this to my attention. Access to a json parser (which simplejson provides) is the only reason why the script was placed in WebKitTools. One could simply check in a second copy of simplejson under WebCore to fix Apple's production build.
The script need not require simplejson, any way to parse json files will do. Other solutions including moving HTMLEntityNames.json away from json, or writing our own json-like parser are possible. When the script was written originally by Adam, he used simplejson. I assume he chose simplejson because it was already used by many other of our scripts and already checked into our repository.
(In reply to comment #3) > The 3rd party python module "simplejson" is already provided in WebKitTools, which is why the tool was placed there. When I placed the tool there, I was not aware of Apple's build limitation. Thank you for bringing this to my attention. It’s always been the case in the Mac OS X and Windows build systems that individual projects cannot access files outside of their source directory. This is precisely the reason why delightful things such as forwarding headers exist. > Access to a json parser (which simplejson provides) is the only reason why the script was placed in WebKitTools. One could simply check in a second copy of simplejson under WebCore to fix Apple's production build. Beyond being obviously unpleasant, I’m not sure that this would work. One further constraint is that it’s not acceptable to modify the source tree in any way. Python writes out .pyc files for modules that it imports, which I believe would violate that constraint.
(In reply to comment #4) > The script need not require simplejson, any way to parse json files will do. Other solutions including moving HTMLEntityNames.json away from json, or writing our own json-like parser are possible. When the script was written originally by Adam, he used simplejson. I assume he chose simplejson because it was already used by many other of our scripts and already checked into our repository. Looking at the input file it seems like JSON is overkill for this file. It’s basically a list of names and corresponding values. Something simple like CSV would be sufficient as far as I can see.
Created attachment 65787 [details] Something like this.
We can use whatever input format is most convenient. I used JSON just because it's easy to work with in most modern languages.
So easy that it requires you install a third-party module… I’ll flag this patch for review shortly. I’m waiting on a couple of full world builds to complete to ensure that everything builds correctly.
Yeah, that looks fine. Does that address the pyc issue as well? (Of course, the final version will need to build on other platforms too.)
> So easy that it requires you install a third-party module… simplejson is built into modern versions of Python. We just need the external module to support Mr. Tiger. > I’ll flag this patch for review shortly. I’m waiting on a couple of full world builds to complete to ensure that everything builds correctly. That patch is close, but there are a bunch of other build systems that need to call this script too. If you grep for HTMLEntityNames.json, you can see how they work.
Yes, it addresses the .pyc issue as there are no .py modules within the source tree that the script imports.
(In reply to comment #11) > > So easy that it requires you install a third-party module… > > simplejson is built into modern versions of Python. We just need the external module to support Mr. Tiger. JSON support was added in Python 2.6. A third-party module is needed even for Leopard. > > I’ll flag this patch for review shortly. I’m waiting on a couple of full world builds to complete to ensure that everything builds correctly. > > That patch is close, but there are a bunch of other build systems that need to call this script too. If you grep for HTMLEntityNames.json, you can see how they work. Yep, I’m aware of that.
> Yes, it addresses the .pyc issue as there are no .py modules within the source tree that the script imports. Oh, cool.
BTW, these files just moved into WebCore/html/parser. You'll probably want to work from a revision in the past few minutes.
Also, thanks for working on fixing this issue.
Created attachment 65795 [details] Patch Updated patch.
Comment on attachment 65795 [details] Patch Thanks. WebCore/html/parser/create-html-entity-table:101 + // THIS FILE IS GENERATED BY WebCore/html/create-html-entity-table WebCore/html/create-html-entity-table => WebCore/html/parser/create-html-entity-table If we were more awesome, we'd generate the path automagically
I noticed one further thing that needs to be included in this patch: updating the references to HTMLEntityNames.json in WebCore.xcodeproj. While I’m there I’m going to stop copying the input file in to the framework wrapper, since copying it there is just stupid. I’ll make these changes as I land.
Attachment 65795 [details] did not build on mac: Build output: http://queues.webkit.org/results/3868031
Landed in r66319.