Request #6351

From:xtina: (blue)[personal profile] xtina
Account Type:Paid Account
Dreamwidth:Account Name: [personal profile] xtina
Style: (S2) core: public, i18n: none, i18nc: none, layout: public, theme: public, user: custom,
Email confirmed? Yes
cluster: 8; data version : 9
scheme: tropo-purple; Language: default
Support category:Importer [Previous|Next]
Time posted:Sun, 16 May 2010 19:24:08 GMT (362 weeks ago)
Status:closed (10 points to [staff profile] mark)
Summary:[MrsJ] Comment content not imported.
Original Request:
Yesterday, I imported my journal and comments from ladytabitha@LJ (an old account) to here. The comments came over, but the content apparently didn't. For an example:

http://xtina.dreamwidth.org/836254.html?format=light

http://ladytabitha.livejournal.com/43439.html?format=light

The number of comments, the commenters, and the nesting are all fine, but there is no content.

Would starting a new import job, and just selecting "comments", be anything like a good idea?

Thankee!
Diagnostics: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 ( )
xtina[personal profile] xtina - Xtina
Comment (#20996)
Posted: Sun, 16 May 2010 19:28:30 GMT (362 weeks ago)

...you know, if I would only read things first...

I have a message in my Inbox:

  Your comments have failed to import.
  Error fetching comment body data from server.

  This was failure #1.

Unfortunately, it doesn't say why. Alas!

The FAQ states that a reimport will not create duplicate entries, but it doesn't mention anything about duplicate comments. I'm idly considering testing this.

chemicallace[personal profile] chemicallace - Chemical Lace
Answer (#21014)
Posted: Mon, 17 May 2010 06:43:27 GMT (362 weeks ago)

Hello Xtina,

Reimporting should not create duplicate comments, but it will bring over any new comments that were made in the mean time. We haven't seen this particular issue before, though, so I am not sure how much of the comment information was imported and if the importer will consider them new. Worst case, we can see if it's possible to have the site administrators remove the imported content so it can be re-imported.

The error itself likely stems from the importer being unable to contact LiveJournal without timing out, so it decides that it's not worth continuing the import. We have a bug open for more informative error messages from the importer so that in the future it won't be quite so confusing.

Should the re-import of the comments not work, we can't see your locked Dreamwidth entry to know exactly what you're seeing. Your description is detailed enough that I have a fairly good idea, but if it's not solved a screenshot or an unlocked example entry would be very helpful.

Please let us know if re-importing doesn't solve the issue and we'll help you keep troubleshooting this.

Best,
ChemicalLace

xtina[personal profile] xtina - Xtina
Comment (#21022)
Posted: Mon, 17 May 2010 08:03:35 GMT (362 weeks ago)

I made a testing journal and did the importing there. As it turns out, the comments weren't duplicated, so that's a yay.

The issue was mostly reproduced in the testing journal, though. "Mostly" because I just imported the comments, rather than comments/tags/entries. I received no error message in my Inbox this time; just one that said the comments were imported.

...and, uh, sorry. I thought I'd unlocked both posts. Turns out I'm only mostly useful. :P

It's likely a timeout issue, because *some* comments came through. I checked the first commentful post of January for each year going back, and I only start receiving contentful comments on 11/29/01 (where the import reached back to April of 2001). Specifically, mid-post here:

http://xtina.dreamwidth.org/900253.html?format=light
http://ladytabitha.livejournal.com/128446.html?format=light

A reimport doesn't change the contentful/-less comments in that post. The 'ladytabitha' journal is defunct, so it shouldn't be getting anything new anyhow, though I may test adding a comment and reimporting, to see what happens.

Another thing I noticed: the empty comments have a created-on date of when I imported them, whereas the full ones have the original time/date.

chemicallace[personal profile] chemicallace - Chemical Lace
Answer (#21136)
Posted: Fri, 21 May 2010 18:37:00 GMT (361 weeks ago)

Hi Xtina,

We're working with the site administrators to figure out what might be causing this, as we haven't had this particular issue before, and how to resolve it. Thank you for all the detailed testing, as it will be a great help. We'll let you know as soon as we have an answer from them.

Best,
ChemicalLace

xtina[personal profile] xtina - Xtina
Comment (#21140)
Posted: Fri, 21 May 2010 23:20:37 GMT (361 weeks ago)

(Clicked the wrong link. :P )

xtina[personal profile] xtina - Xtina
Comment (#21141)
Posted: Sat, 22 May 2010 01:16:52 GMT (361 weeks ago)

I went to export_comments.bml here in DW, and incremented &start_id until I found the spot where it stops failing. The URL that pulls it up (on the off-chance yall can do this for another's journal):

http://www.dreamwidth.org/export_comments.bml?get=comment_body&startid=1880

The comment ID where comment bodies start back up again is #1889. I made screenshots of both DW and the corresponding area in LJ:

http://axisofevil.net/~xtina/img/dw_export.png
http://axisofevil.net/~xtina/img/lj_export.png

Yayfor it being ordered by ID. :P

It's interesting that the LJ ID that corresponds to DW#1889 is #2001.

Just to be thorough:

* Add a new comment to the ladytabitha@LJ post I linked to above.
* Reimport comments only to DW.
* Check the DW export_comments page again.

The comment content starts up at exactly the same place. So adding comments doesn't shift where comments fail out -- seems to be, so far as I can tell, consistently DW IDs 75-1888, or LJ IDs 1-2000.

LJ's comment IDs aren't incrementing precisely by 1 each time, which explains that. So it seems that it's not that it's not getting the first 2000 comments, but rather that it's not getting the first 2000 LJ comment IDs.

And just the bodies, at that.

Goodness. :)

chemicallace[personal profile] chemicallace - Chemical Lace
Answer (#21250)
Posted: Fri, 28 May 2010 03:39:14 GMT (360 weeks ago)

Dear Xtina,

Thank you for all the detailed troubleshooting. I've passed this along to Mark to aid him when he looks at the import logs to get more detailed information on what's going awry.

Best,
ChemicalLace

xtina[personal profile] xtina - Xtina
Comment (#21327)
Posted: Wed, 02 Jun 2010 04:25:28 GMT (360 weeks ago)

Thankee thankee. :) Feel free to ping me to do more testing, if need be -- I test all day anyhow for work, so this isn't out of the ordinary for me.

xtina[personal profile] xtina - Xtina
Comment (#22303)
Posted: Wed, 07 Jul 2010 07:21:53 GMT (355 weeks ago)

I am mostly commenting to say that the issue still occurs, and that I would be willing to provide access to certain DW-oriented folk, should that help matters any.

I'm probably going to forget that this request is open for yet another month, so I do hope this isn't coming across as pressure. I am just all kinds of curious as to why on earth this is happening. (I'm a code junky, though more PHP than Perl, so.)

Thanks!

chemicallace[personal profile] chemicallace - Chemical Lace
Comment (#22848)
Posted: Thu, 29 Jul 2010 16:34:51 GMT (351 weeks ago)

Hello Xtina,

Our backup sysadmin has finally had a chance to look in the logs about this. There's nothing immediately obvious as the cause, other than that it's clearly happening and it's being noted in the logs, but he's going to keep digging and we'll get to the bottom of this (and let you know the in-depth code junky cause).

Best,
ChemicalLace

xtina[personal profile] xtina - Xtina
Comment (#22851)
Posted: Thu, 29 Jul 2010 18:06:15 GMT (351 weeks ago)

Mmm, code-junky causes...

Thank you so much! Again, I'm more than happy to help out if I can, by redoing the import when he wants, or providing the password so he can do it himself, if he wants. (I do have a testing DW account, so he wouldn't even have to worry about b0rking up my active account.)

Whee!

xtina[personal profile] xtina - Xtina
Comment (#25974)
Posted: Thu, 04 Nov 2010 17:42:44 GMT (337 weeks ago)

Hokay, it's been a while, so I tried again. Just importing comments, from ladytabitha, and I get the same issue.

Source:

http://ladytabitha.livejournal.com/43439.html?format=light

Destination:

http://xtina.dreamwidth.org/836254.html?format=light

:(

fu[personal profile] fu - fu
Answer (#26134)
Posted: Fri, 12 Nov 2010 09:37:33 GMT (336 weeks ago)

Hi Xtina,

I'm sorry it's taken so long to go through this! While we still aren't sure why this happened (network blip? comment export temporarily not working? import glitch?), based on the screenshot, it does look like the data is available and we just need to be able to grab it.

The importer right now doesn't edit the comment text on reimport, so I've tweaked it to update the comment text when it runs into a blank comment. The changes will go live with the next code push, tentatively this weekend. I'll ping you when the changes are on the site, and you can then run your comment import again (hopefully with better results).

In case it wasn't a temporary issue -- that is, in case there is something in the data itself (and it's not an issue with fetching the data), this may not yet solve it, but at least we'll know where to dig further. If it turns out to have just been an issue with the initial fetching of the comment bodies, then the blank comment bodies will be fixed.


Best,
Fu

xtina[personal profile] xtina - Xtina
Comment (#26144)
Posted: Fri, 12 Nov 2010 17:18:22 GMT (336 weeks ago)

*facepalm*

If I had more than two brain cells to rub together, it would have occurred to me that of course existing -- or presumed existing -- comments wouldn't be rechecked, as that would eat server time approximately forever. I think I just assumed that the importer would magically know it was a glitch. :P

Thank you so much!

-x

fu[personal profile] fu - fu
Answer (#26180)
Posted: Mon, 15 Nov 2010 05:17:14 GMT (336 weeks ago)

Hi Xtina,

Go ahead and try the import now please, and let us know how it goes!

cheers,
Fu

xtina[personal profile] xtina - Xtina
Comment (#26228)
Posted: Mon, 15 Nov 2010 17:58:07 GMT (336 weeks ago)

Alas, no dice:

http://xtina.dreamwidth.org/836254.html?format=light

http://ladytabitha.livejournal.com/43439.html?format=light

If it would help at all, I could give yall access to the journals, on a for-now basis, so that yall could poke at it more thoroughly?

Stupid question: if I deleted all journal entries in a journal, then reimported, would it do a refreshed import? I have a testing journal created just for this issue, so.

domtheknight[personal profile] domtheknight - domtheknight
Answer (#26229)
Posted: Mon, 15 Nov 2010 18:03:41 GMT (336 weeks ago)

Hi xtina,

If you delete all the entries and then re-import, the importer should pull in all the content again, yes. If you are willing to do that, it could provide some valuable data for testing.

Best,
Dom

xtina[personal profile] xtina - Xtina
Comment (#26240)
Posted: Mon, 15 Nov 2010 22:30:37 GMT (336 weeks ago)

I did a briefer version first, to check.

Steps:

* Deleted the erroring entry: http://xtina.dreamwidth.org/836254.html
* Reimported comments only from ladytabitha@LJ.
* Tracked down the new entry: http://xtina.dreamwidth.org/2353683.html?format=light

Result:

No dice. ?!?

Next up, deleting everything in xtina_testahol (the aforementioned testing account) and reimporting everything, not just comments, to see what happens. I'll report back.

xtina[personal profile] xtina - Xtina
Comment (#26266)
Posted: Tue, 16 Nov 2010 17:54:57 GMT (336 weeks ago)

Steps:

* I deleted the xtina_testahol account, because making a new one was faster.
* I made a new account, xtina_testing.
* I started another import:
** profile data
** journal entries + icon keywords
** journal comments
** tags
** custom security groups
* The importer returned no errors.

The comments still didn't import:

http://xtina-testing.dreamwidth.org/1990102.html?format=light
http://ladytabitha.livejournal.com/43439.html?format=light

Also, I'm showing 7,777 entries in the DW account, but 7,781 in the LJ account. ?! My arms hurt too much right now to go tracking that down, but when I do, I'll let you know.

(I didn't even CHECK that before...)

fu[personal profile] fu - fu
Answer (#26417)
Posted: Tue, 23 Nov 2010 07:42:42 GMT (335 weeks ago)

Hi Xtina,

I wonder -- can you download these two files?

http://www.livejournal.com/export_comments.bml?get=comment_body&startid=1&numitems=1000&props=1
http://www.livejournal.com/export_comments.bml?get=comment_body&startid=1001&numitems=1000&props=1

Those should get us the first 2000 comments, and that's the exact URL that the importer uses (just to be as close to the real input format as possible). Is there anything strange about them? And are you comfortable with either:

a.) running a perl script from the commandline; OR
b.) emailing me the files at fu@dreamwidth.org with all sensitive information edited out?

Hopefully the raw data will get us closer to some idea of what's going on!

Best,
Fu

xtina[personal profile] xtina - Xtina
Comment (#26429)
Posted: Tue, 23 Nov 2010 23:55:26 GMT (335 weeks ago)

I opened the links in new windows -- both of them: "This page can't be viewed except via POST." It apparently hates "props", so I took that off, and it went through fine.

I honestly have no idea what could be going on. I don't see anything strange about them at all. I'll send you comments 1995-2005 (LJ IDs, not DW, obvs), as 2000 is the last comment to be blank, before it starts up again. Then you can be confused, too!

The Perl script would be fine. I don't have the time to go and vouch for comment data for 2000 comments just now, and I like Perl, so. :)

xtina[personal profile] xtina - Xtina
Comment (#26680)
Posted: Thu, 02 Dec 2010 22:26:34 GMT (333 weeks ago)

I tried using ljdump, the Python program, against ladytabitha@LJ:

http://hewgill.com/ljdump/

The comments downloaded fine, so far as I can tell. I could find the comments in the ladytabitha.livejournal.com/43439.html entry just fine.

mark[staff profile] mark - Mark Smith
Answer (#43394)
Posted: Wed, 14 Mar 2012 00:09:03 GMT (267 weeks ago)

Hi,

I'm sorry for the lengthy time before a response here. Usually our support team defers to me on support requests about the importer, and I haven't been watching the queues very well.

Since the last time we touched this thread, the comment importer has basically been rewritten from the ground up. I'd appreciate if you could try it again (if you still want to) and see if it's still having the same problems.

Thank you and, again, I'm sorry for dropping the ball on this.


Best,
    Mark Smith
    Dreamwidth Studios


Go to: previous open request, next open request
Return to the list of open requests.
Back to the Support Area.