Ticket #6805 (closed PLIP: fixed)

Opened 9 years ago

Last modified 6 years ago

Integration of ZODB's blob support into Plone

Reported by: witsch Owned by: witsch
Priority: n/a Milestone: Ongoing
Component: General Version:
Keywords: blobs Cc: limi, rochael, forsberg, lucie, grahamperrin@…

Description

ZODB 3.8 introduced support for blobs (large binary data) out of the box. Plone should utilize this new feature as support for large files without implications on RAM consumption etc has been a long-standing issue. As ZODB 3.8 is not included in Zope 2.10, this feature should probably target Plone 3.5, but compatibility with Plone 3.0 would be nice provided the newer ZODB version is used as well...

Attachments

atct-file-using-blobs.patch Download (1.3 KB) - added by witsch 9 years ago.
the above mentioned patch for convenience…
cmfplone-tests-with-blob-as-atfile-dropin.patch Download (692 bytes) - added by witsch 9 years ago.
the patch mentioned in the last comment for your convenience…

Change History

comment:1 Changed 9 years ago by witsch

  • Owner changed from somebody to witsch
  • Status changed from new to assigned

since i've started working on this during the  piktipi sprint in potsdam and continued during the last couple of days (sprinting remotely with  the folks in boston), i'll take it... :) well, actually the ticket was intended to provide some sort of progress log for myself... :)

comment:2 Changed 9 years ago by witsch

a short status update:

comment:3 Changed 9 years ago by witsch

after checking out the above mentioned branch of ploneout, the obligatory buildout and the necessary changes to zope.conf the following patch can be applied to make the standard "File" content type use ZODB blobs for all new instances:

  • products/ATContentTypes/content/file.py

     
    3333 
    3434from Products.Archetypes.atapi import Schema 
    35 from Products.Archetypes.atapi import FileField 
    3635from Products.Archetypes.atapi import FileWidget 
    37 from Products.Archetypes.atapi import PrimaryFieldMarshaller 
    3836from Products.Archetypes.atapi import AnnotationStorage 
    3937from Products.Archetypes.BaseContent import BaseContent 
     
    5250from Products.ATContentTypes import ATCTMessageFactory as _ 
    5351 
     52from Products.ATBlobField.field import BlobField, BlobMarshaller 
     53 
    5454from Products.validation.validators.SupplValidators import MaxSizeValidator 
    5555from Products.validation.config import validation 
     
    6262 
    6363ATFileSchema = ATContentTypeSchema.copy() + Schema(( 
    64     FileField('file', 
     64    BlobField('file', 
    6565              required=True, 
    6666              primary=True, 
     
    7474                        label=_(u'label_file', default=u'File'), 
    7575                        show_content_type = False,)), 
    76     ), marshall=PrimaryFieldMarshaller() 
     76    ), marshall=BlobMarshaller() 
    7777    ) 
    7878finalizeATCTSchema(ATFileSchema) 

Changed 9 years ago by witsch

the above mentioned patch for convenience...

comment:4 follow-up: ↓ 5 Changed 9 years ago by drjnut

It doesn't work for small files. I didn't do many tests but >1MB works.

comment:5 in reply to: ↑ 4 ; follow-up: ↓ 6 Changed 9 years ago by witsch

Replying to drjnut:

It doesn't work for small files.

yes, for very small files the following error occurs:

Traceback (innermost last):
  Module ZPublisher.Publish, line 119, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 42, in call_object
  Module Products.CMFFormController.FSControllerPageTemplate, line 90, in __call__
  Module Products.CMFFormController.BaseControllerPageTemplate, line 28, in _call
  Module Products.CMFFormController.ControllerBase, line 232, in getNext
   - __traceback_info__: ['id = base_edit', 'status = success', 'button=None', 'errors={}', 'context=<DemoFile at blob.2007-08-20.0582199210>', 'kwargs={}', 'next_action=None', '']
  Module Products.CMFFormController.Actions.TraverseTo, line 38, in __call__
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 42, in call_object
  Module Products.CMFFormController.FSControllerPythonScript, line 104, in __call__
  Module Products.CMFFormController.Script, line 145, in __call__
  Module Products.CMFCore.FSPythonScript, line 140, in __call__
  Module Shared.DC.Scripts.Bindings, line 313, in __call__
  Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec
  Module Products.CMFCore.FSPythonScript, line 196, in _exec
  Module None, line 1, in content_edit
   - <FSControllerPythonScript at /foo/content_edit used for /foo/blob.2007-08-20.0582199210>
   - Line 1
  Module Products.CMFCore.FSPythonScript, line 140, in __call__
  Module Shared.DC.Scripts.Bindings, line 313, in __call__
  Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec
  Module Products.CMFCore.FSPythonScript, line 196, in _exec
  Module None, line 13, in content_edit_impl
   - <FSPythonScript at /foo/content_edit_impl used for /foo/blob.2007-08-20.0582199210>
   - Line 13
  Module Products.Archetypes.BaseObject, line 655, in processForm
  Module Products.Archetypes.BaseObject, line 644, in _processForm
   - __traceback_info__: (<DemoFile at /foo/blob.2007-08-20.0582199210>, <Field file(file:rw)>, <bound method DemoFile.setFile of <DemoFile at /foo/blob.2007-08-20.0582199210>>)
  Module Products.Archetypes.utils, line 160, in mapply
  Module Products.Archetypes.ClassGen, line 76, in generatedMutator
  Module Products.ATBlobField.field, line 105, in set
  Module Products.ATBlobField.adapters.fileupload, line 24, in feed
AttributeError: FileUpload instance has no attribute 'name'

I didn't do many tests but >1MB works.

the limit is actually 1000 bytes. as soon as the file size is 1001 or bigger, everything works as expected. the "problem" here is that python's  cgi module handles small files differently, as can be seen in the __write function at line 691ff in cgi.py (in python 2.4.4):

def __write(self, line):
    if self.__file is not None:
        if self.__file.tell() + len(line) > 1000:
            self.file = self.make_file('')
            self.file.write(self.__file.getvalue())
            self.__file = None
    self.file.write(line)

i'll try to add a workaround for this soon...

comment:6 in reply to: ↑ 5 Changed 9 years ago by witsch

Replying to witsch:

i'll try to add a workaround for this soon...

should be fixed with  r47633

comment:7 follow-up: ↓ 8 Changed 9 years ago by witsch

a short sprint progress report: there are two flavours of blob buildouts now for your convenience, both of which using the new blob integration package:

comment:8 in reply to: ↑ 7 Changed 9 years ago by witsch

Replying to witsch:

a short sprint progress report: there are two flavours of blob buildouts now for your convenience, both of which using the new blob integration package:

for your convenience here are the commands to run these buildout flavours in order to get a running zope instance:

  • for the flavour based on ploneout, i.e. targeted mainly to developers
    svn co https://svn.plone.org/svn/plone/plone.app.blob/buildouts/ploneout
    python bootstrap.py
    ./bin/buildout
    ./bin/instance fg
    
  • for the flavour based on plone.recipe.plone, i.e. targeted mainly to integrators
    svn co http://svn.plone.org/svn/plone/plone.app.blob/buildouts/plone-3.0
    python bootstrap.py
    ./bin/buildout
    ./bin/instance fg
    

comment:9 follow-up: ↓ 10 Changed 9 years ago by witsch

short status update: the above mentioned two flavours of blob buildouts don't use the  special branch of zope 2.10 anymore, but work with the regular zope2 tarballs instead (with some monkey patching involved, of course :))

so, to try to use the integration in your existing plone 3.0 buildout (based on  plone.recipe.plone) the following diff to your buildout.cfg as well as including the externals for PluggableAuthService and plone.recipe.zope2instance to fix pending bugs in these packages should be enough:

  • buildout.cfg

    old new (this hunk was shorter than expected) 
    1414 
    1515develop = 
     16    src/plone.recipe.zope2instance 
    1617 
    1718[plone] 
     
    2527recipe = plone.recipe.zope2instance 
    2628zope2-location = ${zope2:location} 
     29blob-storage = var/blobstorage 
    2730user = admin:admin 
    2831http-address = 8080 
     
    3336    ${buildout:eggs} 
    3437    ${plone:eggs} 
     38    plone.app.blob 
     39    archetypes.schemaextender 
     40    Products.contentmigration 
    3541 
    3642zcml =  
     43    plone.app.blob 
     44    archetypes.schemaextender-overrides 
     45    Products.contentmigration 
    3746 
    3847products = 

comment:10 in reply to: ↑ 9 Changed 9 years ago by witsch

Replying to witsch:

so, to try to use the integration in your existing plone 3.0 buildout (based on  plone.recipe.plone) the following diff to your buildout.cfg as well as including the externals for PluggableAuthService and plone.recipe.zope2instance to fix pending bugs in these packages should be enough:

actually, it's still necessary to include plone.app.blob as a develop egg as well, since i haven't made a release yet... :)

comment:11 follow-ups: ↓ 12 ↓ 13 Changed 9 years ago by klm

i'm blocked trying to build both the ploneout and plone-3.0 buildouts, with this error:

[...]
Getting distribution for 'zdaemon>=1.4a2,<1.4.999'.
While:
  Installing instance.
  Getting distribution for 'zdaemon>=1.4a2,<1.4.999'.
Error: Couldn't find a distribution for 'zdaemon>=1.4a2,<1.4.999'.

i've tried fiddling with the recipe's zdaemon version (and that of ZConfig, which was getting a similar error), but cannot arrive at one that both builds successfully and runs:

  • 'zdaemon>=1.4a2,<1.9.999' cannot be satisfied
  • 'zdaemon>=1.4a2,<2.0' yields zdaemon-20a6, but then trying to start the system results in an error in zdctrl.py:
    "AttributeError: 'dict' object has no attribute 'mapping'".)
    

(that's with a ZConfig entry 'ZConfig >=2.4a2,<3.0', which yields ZConfig 2.5)

so i'm stuck. (i'm working with an earlier version of plone.app.blob that i got working as a zeocluster, and wanted to release instructions for setting up the cluster with the current configuration, but can't until i can get past this.)

comment:12 in reply to: ↑ 11 ; follow-up: ↓ 15 Changed 9 years ago by klm

Replying to myself - klm:

i'm blocked trying to build both the ploneout and plone-3.0 buildouts, with this error:

i've succeeded in using the plone.recipe.plone method, instead, to get a working instance with blobs. being unacquainted with buildouts, there was a lot i had to infer to do so. to help those in a similar predicament (because the more exercising blobs, the better), and to get corrections where i missed an easier way, here's what i did:

  • use the  plone.org buildout tutorial instructions to install ZopeSkel / paster, if you don't already have it. (see  the prerequesites and  creating a buildout... sections).
  • use paster to create a plone3_buildout: paster create -t plone3_buildout Plone-3_0_3-Blobs
  • adjust the buildout, per andi's instructions in this issue (but there's a lot of inferrence to do here):
    • apply the diff suggested in comment 9
    • add src/plone.app.blob to the develop egg section, as sorta suggested in comment 10
    • do svn checkouts of PluggableAuthService, plone.recipe.zope2instance, and plone.app.blob, as implied in the commentary of comment 9. i got the svn checkout addresses from the linked-to EXTERNALS.txt files - i wouldn't be surprised if i could have somehow used those files instead of doing manual checkouts - there's a lot layers involved here.
               cd products
               svn co svn://svn.zope.org/repos/main/Products.PluggableAuthService/branches/1.5/Products/PluggableAuthService PluggableAuthService
               cd ../src
               svn co https://svn.plone.org/svn/plone/plone.app.blob/trunk plone.app.blob
               svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/trunk plone.recipe.zope2instance
      
  • do the bootstrap and buildout:
        python2.4 bootstrap.py
        bin/buildout -v
    

then you'll be able to run as a standalone instance, and when you create a plone site, select the plone.app.blob extension profile so blobs are an available content type. (it's not hard to substitute them for regular files, using the portal_types tool. Images and audio/video are yet to be done.) it looks like getting a zeo cluster going will take similar measures as it did for the ploneout or plone-3.0 integration packages - i plan to test that and make the info available, soon.

comment:13 in reply to: ↑ 11 ; follow-up: ↓ 14 Changed 9 years ago by witsch

Replying to klm:

i'm blocked trying to build both the ploneout and plone-3.0 buildouts, with this error:

hi ken,

that's probably my fault. i was just gonna re-try things with a fresh checkout and about to move my ~/.buildout/default.cfg out of the way when i noticed i have the following line in there:

index = http://download.zope.org/ppix

that's very likely the reason buildout found the eggs for me, but not generically, of course. i'm just double checking atm...

comment:14 in reply to: ↑ 13 Changed 9 years ago by witsch

Replying to witsch:

that's very likely the reason buildout found the eggs for me, but not generically, of course. i'm just double checking atm...

yep, r18085 should fix this problem, so "svn up" please. and sorry about that — rather stupid oversight on my part. actually, i was wondering a bit why merging hanno's r18009 (in r18045) wouldn't cause any of the egg to be not found as i was sort of expecting... :)

comment:15 in reply to: ↑ 12 ; follow-ups: ↓ 16 ↓ 84 Changed 9 years ago by witsch

Replying to klm:

i've succeeded in using the plone.recipe.plone method, instead, to get a working instance with blobs. being unacquainted with buildouts, there was a lot i had to infer to do so.

sorry for the hassle again, ken. like i said, that was a pretty silly mistake to do... :(

to help those in a similar predicament (because the more exercising blobs, the better),

+1 and thanks for the workaround — i hope it shouldn't be necessary to go through all these manual steps anymore, though *duck*. i mean, that was the basic idea of providing the buildouts in the first place, but of course this is still badly lacking documentation...

however, i hope to be able to do a first release to the cheeseshop soon (i.e. tomorrow), which will tell about what to do to get blobs working in a more structured manner.

  • use paster to create a plone3_buildout:

yep, the buildout.cfg you're getting from that will contain the additional helper links from r18085.

(it's not hard to substitute them for regular files, using the portal_types tool. Images and audio/video are yet to be done.)

you can also try to point your browser to .../@@migrate-files-to-blobs to migrate all you existing File content to Blob instances, but i'd highly recommend backing up your Data.fs beforehand at this point... :)

comment:16 in reply to: ↑ 15 ; follow-up: ↓ 17 Changed 9 years ago by klm

Replying to witsch:

Replying to klm:

i've succeeded in using the plone.recipe.plone method, instead, to get a working instance with blobs. being unacquainted with buildouts, there was a lot i had to infer to do so.

sorry for the hassle again, ken. like i said, that was a pretty silly mistake to do... :(

to help those in a similar predicament (because the more exercising blobs, the better),

+1 and thanks for the workaround i hope it shouldn't be necessary to go through all these manual steps anymore, though *duck*. i mean, that was the basic idea of providing the buildouts in the first place, but of course this is still badly lacking documentation...

thanks, andi! i'm more confused, though!-) i tried a few things to obtain your new revisions, but apparently don't understand how i'm supposed to get them. i expect that my answer for now is to work with my above-obtained build, and look forward to replacing that with...

however, i hope to be able to do a first release to the cheeseshop soon (i.e. tomorrow), which will tell about what to do to get blobs working in a more structured manner.

cool! thanks...

comment:17 in reply to: ↑ 16 ; follow-up: ↓ 18 Changed 9 years ago by witsch

Replying to klm:

thanks, andi! i'm more confused, though!-)

oh, that's not what i was hoping to read... :)

i tried a few things to obtain your new revisions, but apparently don't understand how i'm supposed to get them.

actually, you should now be able to use one of the sets of 4 commands listed in my previous comment to get a working instance with blob support; however, i've noticed the "cd ploneout" or "cd plone-3.0" are missing (as the second line in each set)

but i reckon you should also be able to keep using your own buildout and just do an

$ svn up

in the main directory of that; and we should also be able to figure things out "interactively" on #plone if nothing else...

comment:18 in reply to: ↑ 17 ; follow-up: ↓ 19 Changed 9 years ago by klm

Replying to witsch:

Replying to klm:

thanks, andi! i'm more confused, though!-)

oh, that's not what i was hoping to read... :)

i tried a few things to obtain your new revisions, but apparently don't understand how i'm supposed to get them.

actually, you should now be able to use one of the sets of 4 commands listed in my previous comment to get a working instance with blob support; however, i've noticed the "cd ploneout" or "cd plone-3.0" are missing (as the second line in each set)

ah, that was the source of my confusion. i was thinking that the paster of plone3_buildout was the preferred method, and was reluctant to head back to the svn checkouts - but should have realized that's the expected thing to do (and that the paster buildout uses released, not frontier, versions). using an svn checkout will be good - thanks for the clarification.

comment:19 in reply to: ↑ 18 Changed 9 years ago by witsch

Replying to klm:

ah, that was the source of my confusion. i was thinking that the paster of plone3_buildout was the preferred method,

well yes, it is. but you'd only really want to use paster to create your own buildouts — in this case i've already done that for you, so you just need to check it out and run it. that was the basic idea anyway... :) but as a side note, the plone-3.0 buildout was created using paster, of course (see r17624).

and was reluctant to head back to the svn checkouts -

right, but in this case the modified (to the needs for blob support) buildout is provided for your convenience in subversion again. that saves you from creating a fresh buildout (with paster) and applying all the needed modifications (as you're now familiar with :)) as i've already set that up for you.

so instead of having to do several svn checkouts like in the past, you only need one now — essentially you're just getting buildout.cfg and bootstrap.py with the plone.recipe.plone-based flavour. that'd be the ideal scenario anyway — atm the buildout does need some svn:externals, because of some bugfixes that haven't been released in new versions yet. and the egg of plone.app.blob itself is also still missing, of course...

what might have been a little confusing, though, was my comment about how to integrate blob support into your own buildouts. that was meant for people with already existing and customized buildouts, i.e. those who'd like to give blob support a spin on an existing project.

but should have realized that's the expected thing to do (and that the paster buildout uses released, not frontier, versions). using an svn checkout will be good - thanks for the clarification.

yep, atm the checkout's easier, because it automatically gets those still needed and yet unreleased versions for you via svn:externals. once they're released things would ideally boil down to adding the plone.app.blob egg to your buildout.cfg, which would make using paster just as simple again...

comment:20 Changed 9 years ago by witsch

status update: since r18201 it's now actually possible to run the CMFPlone tests using the new blob content type as a drop-in replacement for ATFile with a blob-enabled zodb storage (in the testrunner). to do this you need to patch plone's test base class as follows:

  • products/CMFPlone/tests/PloneTestCase.py

     
    55# $Id$ 
    66 
     7from plone.app.blob import db   # needs to be imported first to set up ZODB 
    78from Products.PloneTestCase.ptc import * 
    89 
     
    1315classImplements(PloneSite, ITestCasePloneSiteRoot) 
    1416 
    15 setupPloneSite(extension_profiles=['Products.CMFPlone:testfixture']) 
     17from plone.app.blob.tests import setupPackage 
     18setupPackage() 
     19 
     20setupPloneSite(extension_profiles=['Products.CMFPlone:testfixture', 'plone.app.blob:testing']) 
    1621 

i'll try to make this somewhat more convenient, but only after the tests pass again... :)

Changed 9 years ago by witsch

the patch mentioned in the last comment for your convenience…

comment:21 Changed 9 years ago by witsch

another update: all CMFPlone tests are passing after r18220... time to go to bed! :)

comment:22 follow-up: ↓ 23 Changed 9 years ago by klm

looks like a file checkin may be missing, andi - trying to start with a buildout from yesterday evening and this afternoon yeilds (a long traceback ending in):

IOError: [Errno 2] No such file or directory: '/home/klm/Servers/Builds/Ploneout+Blobs/addons/archetypes.schemaextender/archetypes/schemaextender/overrides.zcml'

creating a trivial overrides.zcml in the indicated location (with just the <configure> element) enables me to start - but likely isn't the right content.

comment:23 in reply to: ↑ 22 Changed 9 years ago by witsch

Replying to klm:

looks like a file checkin may be missing, andi -

that's because martin's already made the  change to  archetypes.schemaextender we had discussed, but i had not noticed and changed my buildout accordingly yet... r18233 should do this, so an "svn up" should get you going again.

creating a trivial overrides.zcml in the indicated location (with just the <configure> element) enables me to start - but likely isn't the right content.

actually it is, since it's gone now, which is sort of like empty... :) but now the previously empty and therefore not needed configure.zcml has some content and needs to be included.

comment:24 Changed 9 years ago by witsch

  • Cc optilude added

update: after r18236 the buildout based on  plone.recipe.plone is not quite svn:externals-free yet (i.e. "egg-only"), but slowly getting there... perhaps optilude will have some spare time to make a new alpha release of  archetypes.schemaextender soon enough? :)

comment:25 follow-up: ↓ 26 Changed 9 years ago by klm

as threatened, i've created  a web page describing how to use andi's work to build a blob-aware plone site. further, it details how to adjust the buildout and do some manual editing of the result to create a working, blob-aware plone cluster (gotta have zeo:-). i think this will help identify recipe changes to get that working without the edits.

i plan to run through the instructions again to shake out typos and omissions. problem reports and suggestions are welcome...

comment:26 in reply to: ↑ 25 ; follow-up: ↓ 28 Changed 9 years ago by witsch

Replying to klm:

as threatened, i've created  a web page describing how to use andi's work to build a blob-aware plone site.

heh, i've read an early version i think friday night or so, then reloaded the page twice during the weekend, one time giving me "last edited 10 seconds ago", the second "last edited 6 minutes ago", so it sure looked like you were still working on it.

unfortunately, i haven't gotten around to catch up with the updates today, but then again atm it's "last edited 14 minutes ago", so i'm not sure if i should wait a little longer — could you perhaps let me know once you think it's converging to 1.0? :)

i think this will help identify recipe changes to get that working without the edits.

yep, it will, and that's so helpful. thanks for putting this together, ken! oh, and btw, i didn't mention above that the early draft i've read already looked pretty good as a blob howto, and a very detailed one so... :)

comment:27 Changed 9 years ago by witsch

  • Cc optilude removed

oh, and i guess martin's got the message, so i'll reduce his spam again (somewhat)... :)

comment:28 in reply to: ↑ 26 ; follow-ups: ↓ 29 ↓ 30 Changed 9 years ago by klm

Replying to witsch:

Replying to klm:

as threatened, i've created  a web page describing how to use andi's work to build a blob-aware plone site. [...]

unfortunately, i haven't gotten around to catch up with the updates today, but then again atm it's "last edited 14 minutes ago", so i'm not sure if i should wait a little longer — could you perhaps let me know once you think it's converging to 1.0? :)

heh - i should have said, that was my plan.

i just got running with effective-user working, and that's the last of the problems in the things i can think to exercise. i may have scrambled the instructions a bit, in the process of shaking out the later problems - we can handle that as it emerges.

i think this will help identify recipe changes to get that working without the edits.

yep, it will, and that's so helpful. thanks for putting this together, ken! oh, and btw, i didn't mention above that the early draft i've read already looked pretty good as a blob howto, and a very detailed one so... :)

i'm easily confused by ambiguity, so i'm a little extreme about trying to be clear when i write. hope i didn't overdo it.

anyway, i'd say it's cooked. ask when you hit unclear spots...

comment:29 in reply to: ↑ 28 ; follow-up: ↓ 31 Changed 9 years ago by klm

Replying to klm:

Replying to witsch:

Replying to klm:

as threatened, i've created  a web page describing how to use andi's work to build a blob-aware plone site. [...]

unfortunately, i haven't gotten around to catch up with the updates today, but then again atm it's "last edited 14 minutes ago", so i'm not sure if i should wait a little longer — could you perhaps let me know once you think it's converging to 1.0? :)

heh - i should have said, that was my plan.

oh, yeah - i should also have said, i added little "status" notes to the contents list in the intro, at top. i plan to add notes indicating what changed if i discover more things in the already established stuff. conversely, i'll add notes for the later sections (arranging for new files and images to be blob-based, and implementing blob-based video and audio, once i start filling them in. that might be within a few days.

comment:30 in reply to: ↑ 28 Changed 9 years ago by witsch

Replying to klm:

Replying to witsch:

oh, and btw, i didn't mention above that the early draft i've read already looked pretty good as a blob howto, and a very detailed one so... :)

i'm easily confused by ambiguity, so i'm a little extreme about trying to be clear when i write. hope i didn't overdo it.

no, don't get me wrong — i think that's a good thing!

anyway, i'd say it's cooked. ask when you hit unclear spots...

alright, it looks like i won't have time today as i can only work for about 2.5 hours and i want to get the fixes done, but i'll have a look tomorrow...

comment:31 in reply to: ↑ 29 ; follow-up: ↓ 32 Changed 9 years ago by witsch

Replying to klm:

conversely, i'll add notes for the later sections (arranging for new files and images to be blob-based, and implementing blob-based video and audio, once i start filling them in.

the former should be really easy, at least for files. there's a gs profile modifying the "File" FTI so that the new blob content type is used instead. it's currently registered in a way so that it's only available for testing, cause this is what i'm using it for, but it'd be really easy to change that so that one could just apply the profile via /portal_setup and get blobs instead of atfiles. what do you think?

comment:32 in reply to: ↑ 31 ; follow-up: ↓ 37 Changed 9 years ago by klm

Replying to witsch:

Replying to klm:

conversely, i'll add notes for the later sections (arranging for new files and images to be blob-based, and implementing blob-based video and audio, once i start filling them in.

the former should be really easy, at least for files. there's a gs profile modifying the "File" FTI so that the new blob content type is used instead. it's currently registered in a way so that it's only available for testing, cause this is what i'm using it for, but it'd be really easy to change that so that one could just apply the profile via /portal_setup and get blobs instead of atfiles. what do you think?

sounds good! plus, it clues me in to GenericSetup (though it took a bit to figure out that's what "gs" is). are you thinking about implementing this, soon? if not, i'll try to fill in the blanks, though it'd be great if you'd do so, if it would be easy.

once that's done, i expect that i can easily implement something for images, basing it on what's done for files with subtypes/file.py in the plone.app.blob addon.

after that, i imagine it's only a slightly larger step to do something similar for a new audio/video media content type, crafting a schema that provides for an additional metadata field (a place-holder picture). i already have some working skin templates, adapted from an older version of the plone4artists audio/video type, ATVideo, which provide in-browser players adapted to the underlying audio/video medium type. i've used those templates in my previous iteration, inhibiting most or all the other ATVideo machinery because it's there basically to provide a workaround version of blobs...

that's where i'm thinking of heading with this, and there doesn't seem to have any hard pieces. i do have one hesitation, in a design choice, but that i can investigate that while continuing with this course.

that question concerns why a content-type distinction is made at all between underlying binary media. it's tempting to have just one blob-based file type (perhaps with an additional picture place-holder metadata, as i'd do for audio/video) and have the browser templates discriminate how to present the file - as an image, audio player, video player, animation player (flash, etc), pdf, or non-player file. this seems like the simplest thing, UI wise, putting all the presentation smarts in the presentation machinery. it also seems quite justifiable, conceptually - if you're adding some not-directly-editable media, add it as a media file. moreover, as more players arrive on the scene, the players can be back-fitted to existing files of the corresonding media type.

the only potential drawback i see is for categorizing, eg picture collections versus video collections, etc - but that kind of distinction can be based on file metadata, rather than content type, couldn't it?

that question is the one i've been burning to ask for a while, now, but i haven't known who to ask - have you and or alexander questioned the traditional design of distinct content types (having a file versus image versus video versus audio), and/or is there an obvious reason that i'm missing to go this way?

if there's not an easy answer to this question, i can just go with the current design, trying to find the right place to address the question in parallel.

comment:33 Changed 9 years ago by witsch

quick status update before tackling webdav (and answering ken's comment, but probably not before tomorrow): r18321 adds the patches necessary to run the ATCT tests, so you guys can review and also run them yourselves. thanks to a couple of other fixes the only remaining failures are now in  webdav.txt ...

comment:34 Changed 9 years ago by witsch

r18322:18323 fix the remaining issues, so that now all CMFPlone and ATCT tests pass, yay! a good time to go to bed... :)

comment:35 follow-up: ↓ 38 Changed 9 years ago by klm

the buildout command fails with the updated stuff because of a missing cache-size entry in the [common] section of my base.cfg. i've added the line to  my online version of base.cfg, which you can either re-copy or just make the change by hand and check it in. the buildout then works. (it's great to see the changes in place, and get to exercise them - thanks!) i ran out of time (once comcast got my neighborhoods internet connection fixed) to apply and exercise my instructions all the way through, but will do that next chance i get.

comment:36 follow-up: ↓ 39 Changed 9 years ago by andycat

Will this be available for plone 2.5.x ?

comment:37 in reply to: ↑ 32 Changed 9 years ago by witsch

Replying to klm:

sounds good! plus, it clues me in to GenericSetup (though it took a bit to figure out that's what "gs" is).

sorry :)

are you thinking about implementing this, soon? if not, i'll try to fill in the blanks, though it'd be great if you'd do so, if it would be easy.

should be pretty trivial. in fact, the profile's already there for testing purposes — just not exposed to the user yet...

once that's done, i expect that i can easily implement something for images, basing it on what's done for files with subtypes/file.py in the plone.app.blob addon.

well, just replacing the field in ATImage shouldn't be that hard i suppose, but actually re-creating its behaviour as a subtype will bring up some obstacles, imho. however, i guess it's worth the effort nevertheless, since it'll probably clean up things a lot — just like it did for the ATFile replacement. imho that code is much leaner and also cleaner now and still all tests are passing...

after that, i imagine it's only a slightly larger step to do something similar for a new audio/video media content type, crafting a schema that provides for an additional metadata field (a place-holder picture).

yup, that should actually be quite easy, cause afaik the p4a products rely on ATFile and ATImage as their base. so once the basic sub-types are in shape, integration the p4a layer shouldn't be overly complicated, hopefully... :)

that question concerns why a content-type distinction is made at all between underlying binary media. it's tempting to have just one blob-based file type

right, and that's exactly where we're aiming at for plone 4.0, at least that's what everyone agreed upon in naples... ;) it doesn't make sense to have several content types for binary data, of course.

and have the browser templates discriminate how to present the file - as an image, audio player, video player, animation player (flash, etc), pdf, or non-player file.

basically that's where the sub-typing approach helps: you don't really want to put all of that into the same set of templates — that'd be way too messy and not extensible either. instead what sub-typing does is using dynamic marker interfaces and views/temlates/schema-extensions etc registered for those marker interfaces. then you decide, manually or automatically at upload or in whatever way, which "behaviour" you want and mark the blob item accordingly...

moreover, as more players arrive on the scene, the players can be back-fitted to existing files of the corresonding media type.

right, you'd just add another view in that case.

the only potential drawback i see is for categorizing, eg picture collections versus video collections, etc - but that kind of distinction can be based on file metadata, rather than content type, couldn't it?

sure, and it probably should anyway. otoh there will be the need to categorize on your assigned "content type", i.e. the sub-type or behaviour as we call it, but that's also easily done by indexing the marker interfaces each object holds.

have you and or alexander questioned the traditional design of distinct content types (having a file versus image versus video versus audio),

yes, i think we thought about that... :)

and/or is there an obvious reason that i'm missing to go this way?

to not to go this way you mean, no? but no, there isn't, imho.

comment:38 in reply to: ↑ 35 Changed 9 years ago by witsch

Replying to klm:

the buildout command fails with the updated stuff because of a missing cache-size entry in the [common] section of my base.cfg.

hmm, [common] is using the  plone.recipe.zope2instance recipe, which should have a default for the cache-size, no? or rather for either zodb-cache-size or zeo-client-cache-size.

i've added the line to  my online version of base.cfg,

i didn't find anything with the string "cache" in that page. have you fixed the problem in the meantime or am i missing something?

comment:39 in reply to: ↑ 36 ; follow-up: ↓ 42 Changed 9 years ago by witsch

Replying to andycat:

Will this be available for plone 2.5.x ?

yes, that's planned as the next step after the alpha release, i.e. i'll probably be able to start working on that next week...

comment:40 Changed 9 years ago by witsch

just fyi, the first alpha has been officially released to the  PSC and the  cheeseshop — announcement to follow...

comment:41 follow-up: ↓ 43 Changed 9 years ago by witsch

plus, there are new releases of  archetypes.schemaextender and  plone (yay!), so the buildout finally doesn't need any svn:externals anymore... :)

comment:42 in reply to: ↑ 39 ; follow-up: ↓ 47 Changed 9 years ago by klm

Replying to witsch:

Replying to andycat:

Will this be available for plone 2.5.x ?

yes, that's planned as the next step after the alpha release, i.e. i'll probably be able to start working on that next week...

yay on the alpha release!

i think i understand this priority, but want to be sure i understand - does this mean making the ATFile-based-on-blobs profile (adapted from the one you're using for testing) the default, and implementing images/etc as blob-based subtypes, all will come after the 2.5 implementation? and maybe a good bit later?

i'm trying to get a sense of where to aim my sights - if there's a lot of work to get the subtypes implementation done, maybe i'd better just go with the simpler (i presume:-) blob-instead-of-file for each of the binary content types (file, image, audio/video), and wait for the full subtyping implementation like everyone else. does that sound about right? (the mentions i've seen here and there, of plone 4.0 for the full, subtypes-based implementation suggests as much...)

comment:43 in reply to: ↑ 41 ; follow-ups: ↓ 44 ↓ 45 Changed 9 years ago by klm

  • Type changed from enhancement to Bug

Replying to witsch:

plus, there are new releases of  archetypes.schemaextender and  plone (yay!), so the buildout finally doesn't need any svn:externals anymore... :)

my ploneout-based ( https://svn.plone.org/svn/plone/plone.app.blob/buildouts/ploneout), buildout now fails. this is from a checkout done a few minutes ago, and without any custom tailoring. the error is:

[...]
We have the best distribution that satisfies 'setuptools'.
Picked: setuptools = 0.6c7
While:
  Installing instance.
Error: There is a version conflict.
We already have: archetypes.schemaextender 1.0dev
but plone.app.blob 1.0a1dev-r18422 requires 'archetypes.schemaextender>=1.0b1'.

i'm able to get the buildout to complete by editing addons/plone.app.blob/setup.py so that the install_requires entry for schemaextender looks like:

          'archetypes.schemaextender >=1.0dev',

the resulting, completed build seems to work, blobs and all. still, i am not confident that this is the right fix. any idea what's going on?

comment:44 in reply to: ↑ 43 Changed 9 years ago by klm

  • Type changed from Bug to Enhancement

Replying to klm:

whoops - i shouldn't have changed the type property to bug - i thought that was for the individual comment, not the issue as a whole. so i'm changing it back with this - but the issue is still as described in comment 43.

comment:45 in reply to: ↑ 43 ; follow-up: ↓ 46 Changed 9 years ago by witsch

Replying to klm:

my ploneout-based ( https://svn.plone.org/svn/plone/plone.app.blob/buildouts/ploneout), buildout now fails.

r18427 should fix this for now — i'll investigate later...

comment:46 in reply to: ↑ 45 Changed 9 years ago by witsch

Replying to witsch:

r18427 should fix this for now — i'll investigate later...

i did, and r18435 has the details... but in short, "svn up" should fix things properly now... :)

comment:47 in reply to: ↑ 42 Changed 9 years ago by witsch

Replying to klm:

i think i understand this priority, but want to be sure i understand - does this mean making the ATFile-based-on-blobs profile (adapted from the one you're using for testing) the default, and implementing images/etc as blob-based subtypes, all will come after the 2.5 implementation?

as for the profile, no — r18438 has that change. however, it doesn't make it default, but optional. i'll update the docs asap, but will probably try to look into a few bugs (see TODO.txt) so i can make a second alpha including that change.

apart from that, yes, image support will only come after the backport to plone 2.5, so that might still take a little while depending on how well that goes... sorry.

if there's a lot of work to get the subtypes implementation done, maybe i'd better just go with the simpler (i presume:-) blob-instead-of-file for each of the binary content types (file, image, audio/video),

"blob-instead-of-file" is unlikely to work well for images atm. but you're most welcome to try and report what's missing, of course. this would help a lot, in fact.

the other thing you could do in the meantime (i mean, before images are supported through sub-typing), is replace the primary field of ATImage with the blob-based field, similar to one of the initial comments from above...

(the mentions i've seen here and there, of plone 4.0 for the full, subtypes-based implementation suggests as much...)

well, image support will be ready way before 4.0 — i hope to get this done before or at the snow sprint, which would make it even before 3.1, but we'll have to see...

comment:48 Changed 9 years ago by witsch

status update: the  plip reviews have kept me busy last week, but with r18492 a buildout for plone 2.5 now finally starts up at least...

$ svn co https://svn.plone.org/svn/plone/plone.app.blob/buildouts/plone-2.5 blob-2.5
[...snip...]
Checked out revision 18492.
$ cd blob-2.5
$ python bootstrap.py 
Creating directory '/opt/zope/blob-2.5/bin'.
Creating directory '/opt/zope/blob-2.5/parts'.
Creating directory '/opt/zope/blob-2.5/develop-eggs'.
Generated script '/opt/zope/blob-2.5/bin/buildout'.
$ ./bin/buildout 
Develop: '/opt/zope/blob-2.5/src/plone.app.blob'
Installing plone.
Installing zope2.
Installing productdistros.
Installing fakezope2eggs.
Installing instance.
Generated script '/opt/zope/blob-2.5/bin/instance'.
Generated script '/opt/zope/blob-2.5/bin/repozo'.
Installing zopepy.
Generated interpreter '/opt/zope/blob-2.5/bin/zopepy'.
$ ./bin/instance fg
"/opt/zope/blob-2.5/parts/instance/bin/runzope"
2007-12-18 15:11:26 INFO ZServer HTTP server started at Tue Dec 18 15:11:26 2007
	Hostname: 0.0.0.0
	Port: 8080
[...snip...]
2007-12-18 15:11:49 INFO Zope Ready to handle requests

that said, adding a plone site with the "plone.app.blob" profile doesn't work yet, of course... ;)

comment:49 follow-up: ↓ 50 Changed 9 years ago by rochael

  • Cc rochael added

I'm working on an enhancement to plone.recipe.zope2zeoserver so it can generate blob storage configuration AND work off of ZDaemon and ZODB eggs instead of requiring a zope2 installation.

If any one wants to coordinate, please leave a reply.

I intend to commit my works to a branch for evaluation and then merge it back to the recipe trunk.

comment:50 in reply to: ↑ 49 ; follow-ups: ↓ 52 ↓ 53 Changed 9 years ago by klm

Replying to rochael:

I'm working on an enhancement to plone.recipe.zope2zeoserver so it can generate blob storage configuration AND work off of ZDaemon and ZODB eggs instead of requiring a zope2 installation.

that's great to hear! i've been wondering where and how to ask for the zope2zeoserver (and zope2instance) corrections for the problems that my  PloneBlobs instructions workaround. i was thinking about submitting a trac ticket besides this one, but it sounds like you'll be addressing at least some of them. it would probably make sense for you to submit a trac ticket against plone.recipe.zope2zeoserver - that way, whoever feels some responsibility for zope2zeoserver (if anyone) would be alerted, and we would have a place to coordinate.

If any one wants to coordinate, please leave a reply.

all i can offer are the faults and corrections that i noticed - let me know if you need more details or consolidation of what's in the PloneBlobs instructions.

I intend to commit my works to a branch for evaluation and then merge it back to the recipe trunk.

i can probably test that, when you're ready. let us know here what the trac ticket is, if you create one, and include me among the Cc's, if you do.

comment:51 Changed 9 years ago by hannosch

The bug tracker for all the buildout recipes lives at  https://bugs.launchpad.net/collective.buildout/. Feel free to add tickets and patches to it :)

comment:52 in reply to: ↑ 50 Changed 9 years ago by witsch

Replying to klm:

Replying to rochael:

I'm working on an enhancement to plone.recipe.zope2zeoserver so it can generate blob storage configuration AND work off of ZDaemon and ZODB eggs instead of requiring a zope2 installation.

cool! :)

i can probably test that, when you're ready. let us know here what the trac ticket is, if you create one, and include me among the Cc's, if you do.

me too, please.

comment:53 in reply to: ↑ 50 Changed 9 years ago by klm

Replying to klm:

Replying to rochael:

I'm working on an enhancement to plone.recipe.zope2zeoserver so it can generate blob storage configuration AND work off of ZDaemon and ZODB eggs instead of requiring a zope2 installation.

[...] all i can offer are the faults and corrections that i noticed - let me know if you need more details or consolidation of what's in the PloneBlobs instructions.

i've added a few launchpad collective.buildout bug reports:

(while you didn't say you were going to work on plone.receipe.zope2instance, it also needs repair.)

some other things to keep in mind:

  • if i understand christian theune's  Blobs Zero Copy work correctly, much better performance is possible when the blobstorage directory for the server and clients should be the same when they reside on the same machine, so it would be nice if the recipe defaults arranged for that.
  • the var directories need to be created, and the person doing the install needs to be advised about setting the group ownership to that of effective user, if the effective user is set. (it may even be that the user ownership needs to be set - but that requirement may be another bug...)

thanks, hannosch, for the lead on the launchpad topic.

comment:54 follow-up: ↓ 56 Changed 9 years ago by rochael

Arranging for the blob directories to be the same is just a matter of reusing the configuration variable from one section to the other:

[zeoserver]
recipe = plone.recipe.zope2zeoserver
zope2-location = ${zope2:location}
blob-storage = ${buildout:directory}/var/blobstorage
zeo-address = 8100
eggs = ZODB3
...
[instance]
...
blob-storage = ${zeoserver:blob-storage}
...

The recipes need not concern themselves with that, though we should obviously document best practices. Same thing goes for filesystem structure and permissions. Providing examples for those, using "plone.recipe.command", for instance, would go a long way.

comment:55 Changed 9 years ago by rochael

BTW, folks, those interested in buildout+zeo+blob should subscribe to the launchpad issues created by klm above.

comment:56 in reply to: ↑ 54 Changed 9 years ago by klm

Replying to rochael:

Arranging for the blob directories to be the same is just a matter of reusing the configuration variable from one section to the other:

[zeoserver]
recipe = plone.recipe.zope2zeoserver
zope2-location = ${zope2:location}
blob-storage = ${buildout:directory}/var/blobstorage
zeo-address = 8100
eggs = ZODB3
...
[instance]
...
blob-storage = ${zeoserver:blob-storage}
...

in fact, i do that in both my  base.cfg and  buildout.cfg instructions, using indirection through a [common] section. this makes ensuring synchronization between common settings for the server and clients intrinsice, like for eggs and products, which can be (are, in this case) numerous and cumbersome.

having the third [common] section has the added benefit of providing a handle that can be supplemented in derived sections. eg, it's turning out that in buildout.cfg i want my clients and server to use eggs and products that are used in the base.cfg, supplemented by entries for addons - i'm working on incorporating quills into my build. (i wish i knew how to refer to, eg, entries in the [server] section of base.cfg from the [server] section of buildout.cfg, but failed to determine how. that's one of the overwhelmingly many buildout nuances that i haven't got a clue how to track down.)

The recipes need not concern themselves with that, though we should obviously document best practices. Same thing goes for filesystem structure and permissions. Providing examples for those, using "plone.recipe.command", for instance, would go a long way.

is that the recipe used for the unified installer? or is there one? it would be nice to have some canonical recipes - and prominent pointers to them - for anyone looking for places to start in getting acquainted with crafting builds.

anyway, i digress all over the place - i'm very glad you're moving the zope2zeoserver recipe forward!

comment:58 follow-up: ↓ 59 Changed 9 years ago by sunew

Just noticed: there's a wrong security declaration for fixAutoId in field.py. (Haven't got commit rights for plone yet ;) )

comment:59 in reply to: ↑ 58 Changed 9 years ago by witsch

Replying to sunew:

Just noticed: there's a wrong security declaration for fixAutoId in field.py.

fixed in r18648, thanks sune!

comment:60 follow-up: ↓ 61 Changed 9 years ago by rochael

For those not so familiar with using development eggs, to use the branches above, check them out, say, into your src folder and declare them as development eggs:

$ cd src
$ svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2zeoserver/branches/rochael-blobsupport plone.recipe.zope2zeoserver
$ svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/branches/rochael-blobsupport plone.recipe.zope2instance

And in your [buildout] section:

develop =
    src/plone.recipe.zope2zeoserver
    src/plone.recipe.zope2instance
    (...)

comment:61 in reply to: ↑ 60 ; follow-up: ↓ 62 Changed 9 years ago by klm

Replying to rochael:

For those not so familiar with using development eggs, to use the branches above, check them out, say, into your src folder and declare them as development eggs:

$ cd src
$ svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2zeoserver/branches/rochael-blobsupport plone.recipe.zope2zeoserver
$ svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/branches/rochael-blobsupport plone.recipe.zope2instance

And in your [buildout] section:

develop =
    src/plone.recipe.zope2zeoserver
    src/plone.recipe.zope2instance
    (...)

though these measures should be unnecessary with hanno schlichting's commits to the trunk, i did notice a buildout trick that makes this kind of thing easier - worth noting for posterity.

at  http://m.odul.us/2007/10/28/buildout-integrators, matt bowen uses an expression that gets alternative svn checkouts from the net, rather than having to download them manually. for these plone.receipes, you would set find-links entries to the svn address for the alternate repositories and append the name you want to use for the egg on the address - this is what i did in base.cfg:

  find-links =
      [...]
      https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2zeoserver/branches/rochael-blobsupport#egg=plone.recipe.zope2zeoserver                     
      https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/branches/rochael-blobsupport#egg=plone.recipe.zope2instance                       

and add entries for those eggs to the eggs section:

  eggs =
      elementtree
      plone.recipe.zope2instance
      plone.recipe.zope2zeoserver

with these changes (and no changes to the develop = section), the branch versions of the recipes will be automatically used instead of the cheese shop versions. (the branch ones are assigned version increments above the existing recipes, perhaps in lieu of having definite versions of their own.)

the branch versions should be unnecessary now, however, since Hanno Schlichting has checked all the fixes into the trunk. (it's great to see these things falling into place - yay!)

comment:62 in reply to: ↑ 61 Changed 9 years ago by klm

Replying to klm:

Replying to rochael:

For those not so familiar with using development eggs, to use the branches above, check them out, say, into your src folder and declare them as development eggs:

$ cd src
$ svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2zeoserver/branches/rochael-blobsupport plone.recipe.zope2zeoserver
$ svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/branches/rochael-blobsupport plone.recipe.zope2instance

And in your [buildout] section:

develop =
    src/plone.recipe.zope2zeoserver
    src/plone.recipe.zope2instance
    (...)

i was mistaken, below, both about using find-links and about no longer needing to take special measures. just using find-links (at least, the way i was doing) doesn't work without a manual svn checkout, and hanno's trunk commits don't affect the released cheese shop versions - so you do need to fetch the revised recipes, at least until they're released. i recommend you get the trunk versions, since hanno's incorporated leo's fixes and added some.

   cd src
   svn co http://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/trunk plone.recipe.zope2instance
   svn co http://svn.plone.org/svn/collective/buildout/plone.recipe.zope2zeoserver/trunk plone.recipe.zope2zeoserver

and include the develop = src/* lines that leo suggested in comment 60.

this will be unnecessary once the recipes changes are released, which may happen in a week or so.

though these measures should be unnecessary with hanno schlichting's commits to the trunk, i did notice a buildout trick that makes this kind of thing easier - worth noting for posterity.

at  http://m.odul.us/2007/10/28/buildout-integrators, matt bowen uses an expression that gets alternative svn checkouts from the net, rather than having to download them manually. for these plone.receipes, you would set find-links entries to the svn address for the alternate repositories and append the name you want to use for the egg on the address - this is what i did in base.cfg:

  find-links =
      [...]
      https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2zeoserver/branches/rochael-blobsupport#egg=plone.recipe.zope2zeoserver                     
      https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/branches/rochael-blobsupport#egg=plone.recipe.zope2instance                       

and add entries for those eggs to the eggs section:

  eggs =
      elementtree
      plone.recipe.zope2instance
      plone.recipe.zope2zeoserver

with these changes (and no changes to the develop = section), the branch versions of the recipes will be automatically used instead of the cheese shop versions. (the branch ones are assigned version increments above the existing recipes, perhaps in lieu of having definite versions of their own.)

the branch versions should be unnecessary now, however, since Hanno Schlichting has checked all the fixes into the trunk. (it's great to see these things falling into place - yay!)

comment:63 follow-up: ↓ 64 Changed 9 years ago by witsch

quick update regarding the 2.5 backport: things seem to be working ttw and plone.app.blob's tests are passing as of 5 minutes ago; i haven't tried running CMFPlone and ATContentTypes tests yet, but we're getting closer... good night! :)

comment:64 in reply to: ↑ 63 ; follow-up: ↓ 65 Changed 9 years ago by witsch

Replying to witsch:

quick update regarding the 2.5 backport: things seem to be working ttw and plone.app.blob's tests are passing as of 5 minutes ago; i haven't tried running CMFPlone and ATContentTypes tests yet, but we're getting closer... good night! :)

i finally got around to try to run those tests as well (using the patches from the 3.0 buildout as a starting point), and after some minor fixes in r19071 almost all tests are passing now. the one from CMFPlone also fails ootb, but i still have to check the other one from ATCT...

comment:65 in reply to: ↑ 64 Changed 9 years ago by witsch

Replying to witsch:

i finally got around to try to run those tests as well (using the patches from the 3.0 buildout as a starting point), and after some minor fixes in r19071 almost all tests are passing now. the one from CMFPlone also fails ootb, but i still have to check the other one from ATCT...

looking into things again it turns out the two failing tests from ATCT,

Failure in test test_tool_auto_migration (Products.ATContentTypes.tests.test_atfile.TestSiteATFile)
Traceback (most recent call last):
  File "/opt/zope/share/zope/Zope-2.9.8-final/lib/python/Testing/ZopeTestCase/profiler.py", line 98, in __call__
    testMethod()
  File "/opt/zope/sprint/blob-2.5/parts/plone/ATContentTypes/tests/atcttestcase.py", line 298, in test_tool_auto_migration
    self.failUnlessEqual(obj.meta_type, self.meta_type)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 333, in failUnlessEqual
    raise self.failureException, \
AssertionError: 'ATBlob' != 'ATFile'

and

Failure in test test_title (Products.ATContentTypes.tests.test_atfile.TestATFileFields)
Traceback (most recent call last):
  File "/opt/zope/share/zope/Zope-2.9.8-final/lib/python/Testing/ZopeTestCase/profiler.py", line 98, in __call__
    testMethod()
  File "/opt/zope/sprint/blob-2.5/parts/plone/Archetypes/tests/test_baseschema.py", line 120, in test_title
    self.failUnless(isinstance(field.storage, AttributeStorage))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 309, in failUnless
    if not expr: raise self.failureException, msg
AssertionError

are really okay to have: for the first one the meta types obviously differ, but they also should. the second test expects the 'title' field to use an AttributeStorage, but plone.app.blob explicitly sets this field's storage to be an AnnotationStorage in plone/app/blob/content.py:37. if that line's commented out, the failure will go away as well.

so, it seems the backport to plone 2.5 was finished with r19071 (or actually even with r18913), at least as far as the automated tests are concerned. of course, it still needs to see some substantial testing in real projects, but assuming our coverage is good enough, there shouldn't be too many problems left, hopefully... :)

comment:66 Changed 8 years ago by forsberg

  • Cc forsberg added

Trying out plone.app.blob, I ran into memory trouble. I opened up #7780 (or should I have reported it on this ticket instead?).

comment:67 follow-up: ↓ 68 Changed 8 years ago by forsberg

Minor problem: Every time I upload a new Blob, I get this on my console (running in fg mode):

Exception exceptions.OSError: (2, 'No such file or directory', '/tmp/tmp0hnTHq') in <bound method _TemporaryFileWrapper.__del__ of <closed file '<fdopen>', mode 'w+b' at 0xb1aac188>> ignored

comment:68 in reply to: ↑ 67 Changed 8 years ago by witsch

Replying to forsberg:

Minor problem: Every time I upload a new Blob, I get this on my console (running in fg mode):

yes, that's a known issue — see the  faq. the good news is that leo made another fix for windows, which also solves this issue. so again, watch out for the next release... ;)

ps: opening another ticket was fine — this is already way too long anyway...

comment:69 follow-up: ↓ 70 Changed 8 years ago by garbas

anybody implementing image support?

comment:70 in reply to: ↑ 69 ; follow-up: ↓ 71 Changed 8 years ago by witsch

Replying to garbas:

anybody implementing image support?

yes, sune (brøndum wøller; "sunew" on irc) has looked into this and wanted to clean up & commit his changes. i'll be picking up his work (or eventually start my own version :)) soon, i.e. hopefully already next week, but in three weeks time at the latest. please also refer to #7822 for soon to come announcements... :)

comment:71 in reply to: ↑ 70 Changed 8 years ago by garbas

Replying to witsch:

Replying to garbas:

anybody implementing image support?

yes, sune (brøndum wøller; "sunew" on irc) has looked into this and wanted to clean up & commit his changes. i'll be picking up his work (or eventually start my own version :)) soon, i.e. hopefully already next week, but in three weeks time at the latest. please also refer to #7822 for soon to come announcements... :)

great ... well i'll try to be online tonight to catch you on irc. i already looked in latest code in trunk and it does not seems to be that hard to implement it also for other content types. well as you said i'll go to #7822 to discuss this more.

comment:72 Changed 8 years ago by lucie

  • Cc lucie added

comment:73 follow-up: ↓ 74 Changed 8 years ago by klm

andy, i recently tried to create a fresh plone.app.blob based buildout, but blobs are failing in the resulting system. i described the problems in ticket [7986] as best i could, including attaching failures in tests of just plone.app.blob. i explored the issue a little, but don't have the time to uncover what's going on, and am worried i won't be able to migrate my existing sites to new installations...

ken

comment:74 in reply to: ↑ 73 ; follow-up: ↓ 75 Changed 8 years ago by klm

darn! my account of the problem is in ticket 7986, not changeset 7986. (i should have previewed before submitting the comment.)

Replying to klm:

andy, i recently tried to create a fresh plone.app.blob based buildout, but blobs are failing in the resulting system. i described the problems in ticket [7986] as best i could, including attaching failures in tests of just plone.app.blob. i explored the issue a little, but don't have the time to uncover what's going on, and am worried i won't be able to migrate my existing sites to new installations...

ken

comment:75 in reply to: ↑ 74 ; follow-ups: ↓ 76 ↓ 78 Changed 8 years ago by witsch

Replying to klm:

darn! my account of the problem is in ticket 7986, not changeset 7986.

:)

i've answered over there: don't use the ploneout-based buildout while 3.1 it in the making. i'll investigate things once it's been released and i've got some time to spare again... %)

comment:76 in reply to: ↑ 75 ; follow-up: ↓ 77 Changed 8 years ago by witsch

Replying to witsch:

i've answered over there: don't use the ploneout-based buildout while 3.1 it in the making. i'll investigate things once it's been released and i've got some time to spare again... %)

s/it in/is in/ (and yes, previewing helps :))

comment:77 in reply to: ↑ 76 Changed 8 years ago by klm

Replying to witsch:

Replying to witsch:

i've answered over there: don't use the ploneout-based buildout while 3.1 it in the making. i'll investigate things once it's been released and i've got some time to spare again... %)

s/it in/is in/ (and yes, previewing helps :))

:-)

comment:78 in reply to: ↑ 75 Changed 8 years ago by witsch

Replying to witsch:

i've answered over there: don't use the ploneout-based buildout while 3.1 it in the making.

for the record, the buildout based on  plone-coredev a.k.a ploneout is discontinued in favor of the new development buildout... if you want to help developing or simply use the "bleeding edge" version of the blob integration, please switch to the latter using

$ svn sw https://svn.plone.org/svn/plone/plone.app.blob/buildouts/devel

from within your buildout directory...

comment:79 Changed 8 years ago by witsch

(In [21562]) adjust the test patches after r21558 and provide a convenience script to run the CMFPlone and ATContentTypes tests using the new blob content type as a drop-in replacement for ATFile in order to make sure its compatible with the original "File" type (see #6805)

comment:80 follow-up: ↓ 81 Changed 8 years ago by klm

the plone.app.blob checkout and build is broken, i believe because contentmigration is no longer available as an (external) checkout, instead it's available as the Products.contentmigration egg. (see  the README-trunk.txt note.)

i believe what's necessary is to remove the Products.contentmigration entry from src/EXTERNALS.txt and add Products.contentmigration to the buildout.cfg [buildout] section's 'eggs' stanza.

i would try to do this on a branch, but don't have access to the main repository. i'll try copying it all to a repository i set up of my own, so i can make the changes and get past the checkout/update block - but i'm hoping that someone with plone.app.blob interest and checkin access can investigate and rectify promptly.

(with the general state of flux as things develop an transition to eggs, the lack of integration of plone.app.blob with the general plone build is increasingly dicey...)

comment:81 in reply to: ↑ 80 ; follow-up: ↓ 82 Changed 8 years ago by witsch

Replying to klm:

the plone.app.blob checkout and build is broken, i believe because contentmigration is no longer available as an (external) checkout, instead it's available as the Products.contentmigration egg.

thanks for pointing this out, ken. i've changed the external in r22200.

i believe what's necessary is to remove the Products.contentmigration entry from src/EXTERNALS.txt and add Products.contentmigration to the buildout.cfg [buildout] section's 'eggs' stanza.

no, i do want the development version in the development buildout of plone.app.blob. however, i still think you should use the  egg-based buildout for testing or production, especially since i've recently started on image support, which could very well introduce new bugs... ;)

(with the general state of flux as things develop an transition to eggs, the lack of integration of plone.app.blob with the general plone build is increasingly dicey...)

like said above, it's probably a good idea to use the  egg-based buildout unless you'd like to participate in the development of the blob integration.

comment:82 in reply to: ↑ 81 ; follow-up: ↓ 83 Changed 8 years ago by klm

Replying to witsch:

Replying to klm:

the plone.app.blob checkout and build is broken, i believe because contentmigration is no longer available as an (external) checkout, instead it's available as the Products.contentmigration egg.

thanks for pointing this out, ken. i've changed the external in r22200.

cool.

i believe what's necessary is to remove the Products.contentmigration entry from src/EXTERNALS.txt and add Products.contentmigration to the buildout.cfg [buildout] section's 'eggs' stanza.

no, i do want the development version in the development buildout of plone.app.blob. however, i still think you should use the  egg-based buildout for testing or production, especially since i've recently started on image support, which could very well introduce new bugs... ;)

that's helpful, andi. i had shifted to basing my cluster build on the simple zeo recipe in the  plone.app.blob product and  pypi entries. apparently that's more along the lines of the development buildout - the plone-3.x egg-based buildout defers much more to the conventional builds, and so is much better for my purposes.

(with the general state of flux as things develop an transition to eggs, the lack of integration of plone.app.blob with the general plone build is increasingly dicey...)

like said above, it's probably a good idea to use the  egg-based buildout unless you'd like to participate in the development of the blob integration.

aok - thanks! i've adopted the plone-3.x build and it works. i'm hitting a problem migrating my existing blob-containing builds, which i'll detail in a separate entry in this issue...

comment:83 in reply to: ↑ 82 Changed 8 years ago by klm

Replying to klm:

Replying to witsch:

like said above, it's probably a good idea to use the  egg-based buildout unless you'd like to participate in the development of the blob integration.

aok - thanks! i've adopted the plone-3.x build and it works. i'm hitting a problem migrating my existing blob-containing builds, which i'll detail in a separate entry in this issue...

it turns out that i'm unsure about the cause of an unpickling problem (seeking Products.ATBlobFile.interfaces.IFileUpload), and suspect it's separate from plone.app.blob. sorry about the false alarm...

comment:84 in reply to: ↑ 15 ; follow-up: ↓ 85 Changed 8 years ago by grahamperrin

Replying to witsch:

you can also try to point your browser to .../@@migrate-files-to-blobs to migrate all you existing File content to Blob instances

Reading that 2007 comment alongside more recent  http://plone.org/products/plone.app.blob and  http://pypi.python.org/pypi/plone.app.blob I assume that
/@@migrate-files-to-blobs
is superseded by
/@@blob-migration

— please, can we confirm? Thanks.

comment:85 in reply to: ↑ 84 Changed 8 years ago by witsch

Replying to grahamperrin:

— please, can we confirm? Thanks.

yes, and no. while the latest version (i.e. trunk, with a new release pending) still has /@@blob-migration (migrating all your "File" content to a new content type "Blob"), you most likely want to use /@@blob-file-migration ("File" to blob-based "File") and /@@blob-image-migration ("Image" to blob-based "Image") instead. this will not introduce an additional content type, but "simply" convert your existing "File" and "Image" content to use blobs (with hopefully no other noticeable changes :)). this is more in line with #7822 as well, but please note that in order to use these, you'll have to apply the "replacement" profile first.

and yes, the docs are outdated in that regard, but the newer version hasn't been properly released yet, either. it will be soon, though.

comment:86 Changed 8 years ago by hannosch

  • Milestone changed from 4.0 to Ongoing

This ticket isn't really tied to any release but an ongoing discussion.

For Plone 4.0 a real PLIP will be required to get this into the core.

comment:87 Changed 7 years ago by jteeters

Using instruction on the product page:  http://plone.org/products/plone.app.blob with buildout, I seem to got it working with Plone 3.2.1 (current latest release).

By "working" I mean, when I add a new item of type "File" in Plone, the uploaded file is stored in the var/blobstorage directory.

However, when I add an attachment (inside an item of type RichDocument which uses SimpleAttachment), the attachment is not stored in blobstorage.

Three questions:

  1. Is it possible to get SimpleAttachment to use blobstorage? If so, what do I need to do?
  1. Is the patch to file products/ATContentTypes/content/file.py (given in comment dated "07/20/07 23:50:07") no longer applicable? I tried making those changes but got the following error when starting the instance:

ImportError: No module named ATBlobField.field

  1. What is the general state of this product? It seems exactly what I need, but there have not been any posts here for a while. Is it mature enough that it's likely to work well or is there some other product I should consider for storing blobs on the file system and not in the zope db? This is for a production site, but a non-critical application.

Thanks!

comment:88 follow-up: ↓ 89 Changed 7 years ago by hoss

I'm unable to get things working with plone 3.2.1 and r24112 of plone.app.blob.

See  http://plone.org/products/plone.app.blob/issues/18/view

Basically, it throws an exception because of a bogus reference to 'targetpath'.

I had tried the 1.0b2 version of the pypi egg for plone.app.blob, but that didn't work either, which is why I went to the svn trunk (r26015/r24112 as of this post).

I get the distinct impression that plone.app.blob has not kept pace with recent plone core updates. The last concrete confirmed version support was plone 3.0.4.

Please advise. Thanks.

comment:89 in reply to: ↑ 88 ; follow-up: ↓ 90 Changed 7 years ago by klm

Replying to hoss:

I'm unable to get things working with plone 3.2.1 and r24112 of plone.app.blob.

See  http://plone.org/products/plone.app.blob/issues/18/view

Basically, it throws an exception because of a bogus reference to 'targetpath'.

I had tried the 1.0b2 version of the pypi egg for plone.app.blob, but that didn't work either, which is why I went to the svn trunk (r26015/r24112 as of this post).

I get the distinct impression that plone.app.blob has not kept pace with recent plone core updates. The last concrete confirmed version support was plone 3.0.4.

for what it's worth, i have been using plone.app.blob with the most recent version of plone short of plone 3.2. i recently updated my buildout and it seems to be working fine, winding up using plone 3.1.7. (i am under a delivery deadline, and this bug report will dissuade me from trying without the <3.2 constraint until afterwards.)

my buildout plone section looks like this:

[plone]
recipe = plone.recipe.plone >=3.1.1,<3.2dev

Please advise. Thanks.

i'm sorry i can't offer more than evidence that it does work with plone 3.1.7, and a suggestion to settle for that until someone takes responsibility for addressing plone 3.2 compatability.

-- ken  http://myriadicity.net/Sundry/PloneBlobs

comment:90 in reply to: ↑ 89 Changed 7 years ago by hoss

Replying to klm:

(i am under a delivery deadline, and this bug report will dissuade me from trying without the <3.2 constraint until afterwards.)

Thanks for the quick reply.

Yeah, if you've got anything serious you're doing, I'd steer clear of 3.2 until this gets addressed. I may have just had something askew that kept plone.app.blob from working, but I was using as 'clean' of a test as I could manage.

My current choice, it would seem, is to either fall back to <3.2, or just go ahead with my deployment without plone.app.blob. I'm leaning towards the latter, since there seems to be good support for migrating instances of the stock objects to blob storage. I'm not going to be accruing a ton of content right away, so I think I can afford to wait a bit until p.a.b catches up with plone.

comment:91 Changed 7 years ago by witsch

(In [26946]) finally tag  3rd beta (refs #6805, #7822)

comment:92 Changed 7 years ago by witsch

fyi, while the  new beta fixes a number of issues, it's also a bit of a "snapshot" release i wanted to get out because of two reasons: firstly  issue 7 was starting to cause serious trouble in a couple of installations — the last release was clearly showing how outdated it had become.

in addition, i won't be available for about two weeks. i'll continue to work on the package afterwards (as it'll be added to a production deployment soon), but didn't want to hold back the recent progress before that break. there'll also be a hopefully comprehensive blog post soon, but for now there's something far more important to do... :)

comment:93 Changed 7 years ago by witsch

(In [29374]) tag  5th beta (refs #6805, #7822)

comment:94 Changed 7 years ago by witsch

(In [30354]) tag  6th beta (refs #6805, #7822)

comment:95 Changed 7 years ago by witsch

  • Keywords blobs added

comment:96 Changed 7 years ago by witsch

  • Priority changed from major to trivial

this still has some questions i need to respond to, but with plip 7822 being almost done and an imminent first alpha of plone 4 it can mostly be considered finished.

in any case, new issues should not be reported here anyway, but filed either in trac or  poi...

comment:97 Changed 7 years ago by witsch

(In [31159]) tag  7th beta (refs #6805, #7822)

comment:98 Changed 7 years ago by witsch

(In [31722]) tag  8th beta (refs #6805, #7822)

comment:99 Changed 7 years ago by witsch

(In [32057]) tag  9th beta (refs #6805, #7822)

comment:100 Changed 7 years ago by witsch

(In [32260]) tag  10th beta (refs #6805, #7822)

comment:101 Changed 7 years ago by witsch

(In [33593]) tag  11th beta (refs #6805, #7822)

comment:102 Changed 6 years ago by grahamperrin

  • Cc grahamperrin@… added

comment:103 Changed 6 years ago by witsch

(In [33691]) the  plone 4.0-specific versions of plone.recipe.zope2instance don't work with python 2.4 since they rely on the  hashlib module — this should hopefully fix the  bot again (refs #6805, #7822)

comment:104 Changed 6 years ago by witsch

(In [34040]) tag  12th beta (refs #6805, #7822)

comment:105 Changed 6 years ago by kleist

Shouldn't we close this one?

comment:106 Changed 6 years ago by limi

  • Status changed from assigned to closed
  • Priority changed from trivial to n/a
  • Resolution set to fixed
  • Type changed from Feature Request to PLIP

Yes, we should — witsch also said that we should track any other issues separately, and I agree.

comment:107 Changed 4 years ago by davisagli

  • Component changed from Infrastructure to General
Note: See TracTickets for help on using tickets.