Request #49206
From:
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Account Type:
Paid Account
Dreamwidth:
Support category:
Time posted:
Wed, 28 Aug 2024 12:33:56 GMT (36 weeks ago)
Status:
Open
Summary:
Search robots don't see my journal?...
Original Request:
Hello!
I suddenly discovered that dreamwidth.org has a problem with visibility for search robots. For example, I added the google-site-verification meta tag to the head section (it's definitely there - I can see it when viewing the page source code), but Google Search Console tells me that it doesn't see any google-site-verification at pilottttt.dreamwidth.org. The same thing happens with Google Analytics: their tracking code is on my pages, and it even works (Google shows statistics), but if I click the "Test your site" button in my Google Analytics account, the message "No Google tag found on your site" appears in response. It seems that the Google robot simply doesn't see the main page of my journal.
I also checked the availability of my journal on webmaster.yandex.ru using their "Server Response Check" tool. I tried different pages (for example, https://pilottttt.dreamwidth.org/424333.html) with different User-Agent values, but Yandex Webmaster responded each time that it received "403 Forbidden". So, for this reason, Yandex search robots also do not see my journal.
It is possible that the visibility issue exists for other search engines as well.
Diagnostics: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:129.0) Gecko/20100101 Firefox/129.0
I suddenly discovered that dreamwidth.org has a problem with visibility for search robots. For example, I added the google-site-verification meta tag to the head section (it's definitely there - I can see it when viewing the page source code), but Google Search Console tells me that it doesn't see any google-site-verification at pilottttt.dreamwidth.org. The same thing happens with Google Analytics: their tracking code is on my pages, and it even works (Google shows statistics), but if I click the "Test your site" button in my Google Analytics account, the message "No Google tag found on your site" appears in response. It seems that the Google robot simply doesn't see the main page of my journal.
I also checked the availability of my journal on webmaster.yandex.ru using their "Server Response Check" tool. I tried different pages (for example, https://pilottttt.dreamwidth.org/424333.html) with different User-Agent values, but Yandex Webmaster responded each time that it received "403 Forbidden". So, for this reason, Yandex search robots also do not see my journal.
It is possible that the visibility issue exists for other search engines as well.
Diagnostics: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:129.0) Gecko/20100101 Firefox/129.0
UPD. I checked: the setting "Attempt to block outside search engines from indexing my journal" is disabled in my journal, and I changed "Searching Your Journal" to "Allow any logged-in user to search my journal" just in case. But this, of course, did not help.
So this setting doesn't work: https://live.staticflickr.com/65535/53955347931_5b0083603e_o.png
UPDUPD: I did a little experiment: I sent GET requests to https://pilottttt.dreamwidth.org/ with different User-Agent values, while changing the "Attempt to block outside search engines from indexing my journal" setting in my profile. You can see the result here: https://live.staticflickr.com/65535/53969234142_0e8f68af9b_o.png, and this result surprised me. Firstly, it confirmed my assumption that this setting does not work. Secondly, it turned out that DreamWidth turns down only Yandex search robots. But then it is unclear why Google does not see its verification tag on my main page. Maybe your system recognizes search robots by some other criteria besides the User-Agent value?
You must log in to answer Support requests.
Go to: previous open request, next open request
Return to the list of open requests.
Back to the Support Area.