In looking at the comments of the last post The Search for Semantic Search, I see there appears to be some interesting interpretations. Let me explain my motives, address any perceived bias and clarify my position.
Alex Iskold wrote about semantic search that we were asking the wrong questions; that it was essentially the root of the problem with semantic search engines, and; they were only capable of handling a narrow range of questions– those requiring inference. Among other things, he also wrote that his question about vocation was unsolvable; impossible, was the term he used. These ideas and the fact that Alex implied Google was a semantic search engine, and inferred that vendors must dethrone Google to be successful, motivated me to blog about it myself.
I was criticized, in the comments, for implying that the so-called “semantic search” capability of these NLP-driven search engines is weak, and due to this they do not really qualify as “semantic search” engines. Actually Kathleen Dahlgren introduced a new name in her comments: “Semantic NLP”. I was also criticized for asking a silly question and for posting my brief analysis of this one single question that Alex said was unsolvable without massively parallel computers.
Of course you cannot judge a search engine by the way it mishandles one or even a few queries. But in this case one natural language question reveals a lot about the semantic acuity of NLP, and the multiple query idea is a kind of strawman argument intended to distract us. It almost proves Alex right as it is misleading.
I do not believe people are motivated to ask wrong questions and I do not believe people ask silly questions to computer search engines while expecting a satisfactory set of results or answers. Nevertheless, when any case fails, the problem or fault does not lie with people. The search engine is supposed to inform them. The fault lies with the computer software for failing to inform. You can try and dismiss it with a lot of hand waving but just like that pesky deer fly — it is going to keep coming back.
While NLP front ends and semantic search engines are the result of millions of dollars in funding and the long work of brainiacs, and while they may be capable enough to parse sentences, build an index or other representation of terms and use some rules of grammar, they are not always accommodating or satisfying. In fact they can be quite brittle or breakable. This means they do not always work. But they do work under the right circumstances in narrowly defined situations. One of the questions here is whether they work well enough to qualify them as “semantic search” engines for English language questions.
Any vendor who comes out in public and claims they are doing “semantic search” should prove it by inferring the significance of the input with sufficient quality and acuity such that the result, or search solution or conclusion, satisfies the evidence and circumstance. This a minimum level of performance. There are tests for this. Many people use a relevance judgment as a measure of that satisfaction as far as any type of search and retrieval method or software is concerned.
With that said, my last post was about debunking the so-called complex query myth not about “testing” the capabilities of any search engine. It was about semantic search and how any search engine solves this single so-called impossible question. There were results, and they were not completely “useless” as I see, on review, that I wrote. I apologize for calling them useless.
Both Cognition and Powerset produced relevant results (with one word) that were more comprehensive than the results Google provides, in my opinion. That is not a natural language process of understanding a sentence though. Having a capacity to look up a word in a dictionary is not the same as the capacity to referentially or inferentially use the concept. In this case, to make some judgments (distinguish the significant relationships, at least) and inform the search process.
This capability to distinguish significant relationships is a key criteria of “semantic” search engines — meaning they should have a capacity to infer something significant from the input and use it. The results of this query tell a different story. You cannot just profess linguistic knowledge, call the question silly and make the reality it represents go away. This kind of problem is systemic.
As far as the so-called “semantic” search engines inferring anything siginificant from this (full sentence case) question (evidence) or circumstance of searching, I treated all the results with equal disaffirmation. What is more; I stand by that as it is supported on its face. If you look at the results of the full sentence case query at Cognition, you will notice that they are essentially the same as those from Powerset.
I reckon this could be because both engines map the parts of speech and terms from the query onto the already prepared terms and sentences from Wikipedia. This “mapping strategy” clearly fails –in this case– for some pretty obvious reasons. Without pointing out all the evidence I collected, I summed those reasons up as a lack of semantic acuity. That seems to have touched a nerve.
So I will get into the details of this below. Let me first take a moment to address the fact that one inquiry reveals all this information. Really it is not just one inquiry. It is one conceptualization. Dozens of questions can be derived by mapping from the concepts associated to these terms of this single question. For example: Where are the best careers today?; Who has the better jobs this year?; Where can I work best for now?; What occupation should I choose given the times?; etc. I tried them all and more with varying degrees of success.
One problem is that NLP practitioners are concerned with sentence structure and search engineers are concerned with indexing terms and term patterns. Either way, the methods lack a conceptual dimension and there is no apparent form of any semantic space for solving the problem. The engines have no sense of time points or coordinated space or other real contexts in which things take place. The absence of semantic acuity is not something that only affects a single inquiry. It will infect many inquiries just as a disease infects its host.
Now that I recognize the problem, if I were challenged to a wager, I would wager that I could easily produce 101 regular English language questions that would demonstrate this affliction. The search engines may produce a solution, except that the results would be mostly nonsense and not satisfying. It would prove nothing more and nothing less than I have already stated. What say you Semantic Cop?
I should mention that I have long suspected that there was a problem mapping NLP onto a search process and I could not put my finger on it. A literature search on evaluations of text retrieval methods will show, in fact, that the value of part of speech processing (in text search and retrieval) has long been regarded as unproven. By taking the time to investigate Alex Iskold’s complex query theory I gained more insight into the nature and extent of this problem. It is not just a problem of finding a definition or synonyms for a given term as some reader’s may infer. Let me explain further.
While Powerset, Cognition and Hakia each had the information that a vocation was a kind of altruistic occupation, and the search circumstance (a hint) that the information seeker could be looking for an occupational specialty or career, they did not really utilize that information. The failure, though, really wasn’t with their understanding of the terms occupation or vocation. Their failure was specifically related to the NLP approach to the search process. That is supported by the fact that these different search products employing NLP fail in the same way.
That should not be taken to mean that the products are bad or useless. Quite to the contrary, the product implementations are really first class productions and they appear to improve the user experience as they introduce new interface techniques. I think NLP technology will continue to improve and will eventually be very useful, particularly at the interface as Alex noted in his post. But does that make them semantic search engines?
Lest I have been ambiguous, let me sum up and clarify by referring back to the original question: Whether you are looking at Powerset, Cognition or Hakia results, they clearly did not understand the subordinate functionality of the terms /best/, /vocation/, /me/ and /now/ in the sentence.
They clearly could not conceptualize ‘best vocation’ or ‘now’– they could only search for those keyword patterns in the index or data structures created from the original sentences. That is not just ‘weak’ semantics that is not semantic search at all. Maybe they “understood” the parts of speech but they did not infer the topic of inquiry nor did they properly map the terms into the search space. Google did not fare any better in this case, but Google does not claim to be a semantic search engine. So where’s the semantics?
By that I mean (for example) that interpreting /now/ from the natural language question ‘what is the best vocation for me now’ as an adverb, does not improve the search result. Treating it as a keyword or arbitrary pattern does not improve the search result. And it demonstrates a clear and present lack of acuity and understanding of the circumstance.
Finding the wrong sense of /now/ and showing it is of dubious value. An inference from /now/ to ( –> ) today, at present, this moment in time, or to this year or age, and using that as evidence leading to an informed conclusion would demonstrate some semantic acuity in this case. Most people have this acumen, these NLP search engines obviously do not–according to the evidence in this case.
The NLP vendors defend this defect by accusing people of not asking the right question in the right way or not asking enough questions. That is like me saying to my wife:
If you want some satisfying information from me you better use a lot of words in your question and it better not be silly. Don’t be too precise and confuse me and don’t use an idiom and expect me to satisfy you. I’ll still claim to understand you. It is you that asks silly questions. That not being enough, you also have to nag me with more long and hard questions before you say my responses are rubbish.
If I should either desire or dare to do that at all, what do you think her response would be? More importantly what do prospects say when you tell them their questions are silly?
I do not need to proceed with a hundred questions when with a dozen or so I have enough evidence to deduce that these NLP-driven search engines are limited when it comes to ** inferring the topic of inquiry **. In some cases they are simply unable to draw on the *significant structures or patterns of input, evidence or circumstance” and produce a suitable solution.
What bothers me is that some of these so-called “semantic search engines” claim to “understand” in general. I did that too, a very long time ago. Yes, I was there in the back of the room at DARPA and NIST meetings and I have been at PARC and the CSLI for presentations. I was challenged then. And it enlightened me. If such claims go unchallenged it will only serve to demean the cultural arts and independent critical thinking and confuse prospects about the capabilities regular people expect of semantic products. I do not wish to lower the bar.
In this instance, and there are many similar cases that could be derived from the semiotic characteristics of this instance, the NLP-driven engines do not show the slightest acumen for inferring the topic of inquiry. I hope the discerning reader sees that it is not just about some synonyms. If they could infer the topic of inquiry, that would demonstrate a little understanding… at least a capacity to learn.
The result, in all such cases, is that these so called “reasoning-engines” and semantic search engines do not lead us to a satisfying consequence or conclusion at all. They have technical explanations such as synonymy and hyponymy for any word, yet, if the software cannot infer the sense of everyday terms, is it even sensible to call the methods “semantic”? Just because the vendors profess linguistic knowledge does not mean their their semantics are any more than just another marketing neologism.
It may be called semantic NLP but that does not qualify as semantic search in my opinion.
